Plasma Documentation

Plasma release: 0.6 "Rechenknecht". This is a beta release, intended for broader testing. This release contains:

• PlasmaFS: filesystem
• PlasmaMapReduce: compute framework
• PlasmaKV: key/value databases

General documents:

• Plasma_release: Release notes
  • Plasma_release_0_1: Old release notes for 0.1
  • Plasma_release_0_2: Old release notes for 0.2
  • Plasma_release_0_3: Old release notes for 0.3
  • Plasma_release_0_4: Old release notes for 0.4
  • Plasma_release_0_5: Old release notes for 0.5
• Plasma_license: License
• Plasma_quickstart: How to get quickly started with Plasma

Transcripts of sample sessions:

• Sess_plasma_install: Build and install of Plasma using plasma_install.sh
• Sess_postgresql_conf: Configure PostgreSQL
• Sess_deploy: Deploy PlasmaFS
• Sess_mr_wordfreq: Run the classic map/reduce job mr_wordfreq
• Sess_mr_cardgames: Run the toolkit-based map/reduce job cardgames

Web resources:

• Plasma homepage
• Frequently asked questions

PlasmaFS Documentation

PlasmaFS is the distributed transactional filesystem. It is implemented in user space, and can be accessed via RPC calls, or via NFS.

• Plasmafs_start: Feature set, theory of operation
• Plasmafs_deployment: Deploying PlasmaFS
• Plasmafs_dn_discovery: Configuring the datanode discovery
• Plasmafs_authentication: Authentication
• Plasmafs_dn_admin: Datanode administration
• Plasmafs_nn_admin: Namenode administration
• Plasmafs_protocol: The PlasmaFS protocol
• Plasmafs_nfs: Configuring NFS
• Plasmafs_and_hdfs: Feature comparison of PlasmaFS and HDFS

PlasmaFS Commands

• Cmd_plasma: The plasma utility
• Cmd_plasmad: The plasmad daemon for datanodes and namenodes
• Cmd_plasma_datanode_init: The plasma_datanode_init utility for initializing datanodes
• Cmd_plasma_admin: The plasma_admin utility for doing namenode administration
• Cmd_nfs3d: The nfs3d daemon for the NFS bridge

PlasmaFS Client Interfaces

Client applications can only link with plasmasupport and plasmaclient.

plasmaclient: RPC client

Plasma_client	Client access to the Plasma Filesystem
Plasma_client_config	Configuration of clients
Plasma_shm	Management of large buffers incl.
Plasma_inodecache	The inodecache stores the metadata found in inodes, namely the inodeinfo struct, and subsets of the block list.
Plasma_filebuf	Buffer for read and write
Plasma_blocks	Helpers for analyzing blocklists
Plasma_netfs	Accessing PlasmaFS via the Netfs.stream_fs interface

These are the mappings of the XDR definition to Ocaml (as used in the client):

Plasma_rpcapi_aux
Plasma_rpcapi_clnt

plasmasupport: Support library

Plasma_rng	Random numbers
Plasma_ug	Users and groups
Plasma_platform	Some C functions

PlasmaFS RPC protocol definition

These are XDR definitions:

• Pfs_types: Types for all RPC definitions
• Pfs_nn_fsys: Filesystem access
• Pfs_nn_coord: Determining the coordinator
• Pfs_nn_dnadmin: Administration of the known datanodes
• Pfs_nn_internal: Internal interfaces used between PlasmaFS RPC servers
• Pfs_datanode: Datanode access
• Pfs_dn_internal: Internal interfaces used between PlasmaFS RPC servers
• Pfs_nfs3: NFS version 3
• Pfs_server: All RPC definitions for server context
• Pfs_client: All RPC definitions for client context

PlasmaFS Internal Server Interfaces

The following interfaces exist only within the server.

pfs_support: Support library

Pfs_db	Utility functions for database accesses
Pfs_auth	Authentication in server context
Pfs_condition	Condition variables
Pfs_pmanage	Manager for potentially persisting objects

These are the mappings of the XDR definition to Ocaml (as used in the server):

Pfs_rpcapi_aux
Pfs_rpcapi_clnt
Pfs_rpcapi_srv

pfs_datanode: Datanode server

Dn_config	The clustername
Dn_store	The directory containing the store.
Dn_shm	Shared memory for transmitting data from the main process to the I/O processes and vice versa
Dn_io	Returns this as factory for inclusion into Netplex_main.startup
Dn_manager	Returns this as factory for inclusion into Netplex_main.startup

pfs_namenode: Namenode server

Nn_config	Configuration of name nodes
Nn_db	Namenode access routines to database
Nn_blocklist	Support module for blocklists
Nn_blockmap	Already allocated blocks of this inode.
Nn_blockmap_box	Fast RPC to the blockmap server via camlboxes
Nn_alloc_types
Nn_alloc_trivial	Block allocation algorithm
Nn_alloc_balanced	Block allocation algorithm
Nn_datanode_ctrl	Manage tickets and push ticket secrets to the datanodes
Nn_datastores	Datastore types
Nn_datastore_news	Getting news about datastores and their availability
Nn_state	State manager: Provide the logical view of the state, and forward changes to the database and to the nameslaves (2-phase commit)
Nn_commit	Two-phase commit protocol
Nn_elect	Election of the coordinator
Nn_dentrycache	The dentrycache checks whether directory metadata is still up to date.
Nn_inodecache	The inodecache checks whether inode metadata is still up to date.
Nn_ug	Users and groups
Nn_monitor	Management of datastores, i.e.
Nn_push	Push updates to the slaves (client)
Nn_slave	The implementation of the namenode component for slave nodes
Nn_coord	The implementation of the namenode component for the coordinator node
Nn_authticket	Management of authentication tickets

pfs_nfs3: NFS server

Nfs3_aux
Nfs3_clnt
Nfs3_srv

Plasma MapReduce

Plasma MapReduce is the compute framework for Map/Reduce-style data transformations. It uses PlasmaFS for storing the data:

• Plasmamr_start: What is Map/Reduce?
• Plasmamr_howto: How to run a Map/Reduce job
• Plasmamr_toolkit: The map/reduce toolkit
• Plasmamr_task_types: The tasks a Map/Reduce job is composed of
• Plasmamr_file_formats: Description of the possible file formats
• Plasmamr_and_hadoop: Feature comparison of PlasmaMR and Hadoop

Plasma MapReduce Commands

• Cmd_plasma_convert: Convert between file formats

Plasma MapReduce Interfaces

mr_framework: Framework

Mapred_io	Utility library for record-based I/O
Mapred_io_bgreader	Background reader for Mapred_io
Mapred_config	Configuration of mapred servers
Mapred_job_config	Extract job configuration, and marshalling
Mapred_def	Defining the map/reduce job
Mapred_split	Splits a line into key and value.
Mapred_sort
Mapred_sorters	Sorter classes
Mapred_taskfiles	Copy files to and from the task nodes
Mapred_tasks	Representation of tasks
Mapred_sched	Scheduler
Mapred_task_exec	Executes this task
Mapred_task_server	Factory for the processor "mapred_task_server"
Mapred_job_exec	Execute a scheduled job by submitting tasks to servers
Mapred_job	Run a map/red job
Mapred_streaming	Support for streaming
Mapred_main	Main program for the task server and job controller, for this single job only
Mapred_stats
Mapred_toolkit	Map/reduce toolkit
Mapred_fields	Access to record fields
Mapred_rfun	Registered functions (low-level)
Mapred_fs	Filesystem abstraction
Mapred_auth	Authentication in server context
Mapred_rpc_aux
Mapred_rpc_clnt
Mapred_rpc_srv

mr_platform: Platform-specific functions

Platform

Platform-specific functions

Plasma MapReduce RPC protocol definition

These are XDR definitions:

• Pmr_task: The RPC definition of the task executor

PlasmaKV Documentation

Key/value databases are just files mapping keys to arbitrarily large values. The files live, of course, inside PlasmaFS. In transactional mode, the databases are ACID-compliant and allow high concurrency.

• Plasmakv_intro: Plasma Key/Value Databases

PlasmaKV commands

• Cmd_plasma_kv: The plasma_kv utility

PlasmaKV Interfaces

Pkv_api

Plasma Key Value API

This web site is published by Informatikbüro Gerd Stolpmann