API Documentation

cloud — PiCloud Interface

Interface for cloud - the clientside module for PiCloud

Sample usage:

import cloud
jid = cloud.call(lambda: 3*3)
>> Returns a job identifier
cloud.result(jid)
>> Returns 9

This will run the function lambda: 3*3 on PiCloud’s cluster, and return the result. Most functions, even user-defined ones, can be passed through cloud.call

For cloud to work, you must first run ‘picloud setup’ in your shell.

Alternatively, you can use the simulator by setting use_simulator to True in cloudconf.py or running cloud.start_simulator()

cloud.call(self, func, *args, **kwargs)

Create a ‘job’ that will invoke func (a callable) in the cloud.

When the job is run, func will be invoked on PiCloud with the specified args and all non-reserved kwargs.

Call will return an integer Job IDentifier (jid) which can be passed into the status and result methods to obtain the status of the job and the job result.

Example:

def add(x, y):
    return x + y
cloud.call(add, 1, 2) 

This will create a job that when processed will invoke add with x=1 and y=2.

See online documentation for additional information.

Reserved special kwargs (see docs for details):

  • _callback:

    A list of functions that should be run on the callee’s computer once this job finishes successfully. Each callback is passed one argument: the jid of the complete job

  • _callback_on_error:

    A list of functions that should be run on the callee’s computer if this job errors.

  • _cores:

    Set number of cores your job will utilize. See http://docs.picloud.com/primer.html#choose-a-core-type/ In addition to having access to more CPU cores, the amount of RAM available will grow linearly. Possible values for _cores depend on what _type you choose:

    • c1: 1
    • c2: 1, 2, 4, 8
    • f2: 1, 2, 4, 8, 16
    • m1: 1, 2, 4, 8
    • s1: 1
  • _depends_on:

    An iterable of jids that represents all jobs that must complete successfully before the job created by this call function may be run.

  • _depends_on_errors:

    A string specifying how an error with a jid listed in _depends_on should be handled:

    • ‘abort’: Set this job to ‘stalled’ (Default)
    • ‘ignore’: Treat an error as satisfying the dependency
  • _env:

    A string specifying a custom environment you wish to run your job within. See environments overview at http://docs.picloud.com/environment.html

  • _fast_serialization:

    This keyword can be used to speed up serialization, at the cost of some functionality. This affects the serialization of both the arguments and return values Possible values keyword are:

    1. default – use cloud module’s enhanced serialization and debugging info
    2. no debug – Disable all debugging features for arguments
    3. use cPickle – Use Python’s fast serializer, possibly causing PicklingErrors

    func will always be serialized by the enhanced serializer (with debugging info).

  • _kill_process:

    Terminate the jobs’ Python interpreter after func completes, preventing the interpreter from being used by subsequent jobs. See Technical Overview for more info.

  • _label:

    A user-defined string label that is attached to the jobs. Labels can be used to filter when viewing jobs interactively (i.e. on the PiCloud website).

  • _max_runtime:

    Specify the maximum amount of time (in integer minutes) jobs can run. If the job runs beyond this time, it will be killed.

  • _os_env_vars:

    List of operating system environment variables that should be copied to PiCloud from your system Alternatively a dictionary mapping the environment variables to the desired values.

  • _priority:

    A positive integer denoting the jobs’ priority. PiCloud tries to run jobs with lower priority numbers before jobs with higher priority numbers.

  • _profile:

    Set this to True to enable profiling of your code. Profiling information is valuable for debugging, and finding bottlenecks. Warning: Profiling can slow your job significantly; 10x slowdowns are known to occur

  • _restartable:

    In the rare event of hardware failure while a job is processing, this flag indicates whether the job can be restarted. By default, this is True. Jobs that modify external state (e.g. adding rows to a database) should set this False to avoid potentially duplicate modifications to external state.

  • _type:

    Choose the type of core to use, specified as a string:

    • c1: 1 compute unit, 300 MB ram, low I/O (default)
    • c2: 2.5 compute units, 800 MB ram, medium I/O
    • f2: 5.5 compute units, 3.75 GB ram, high I/O, hyperthreaded core
    • m1: 3.25 compute units, 8 GB ram, high I/O
    • s1: Up to 2 compute units (variable), 300 MB ram, low I/O, 1 IP per core

    See http://www.picloud.com/pricing/ for pricing information

  • _vol:

    A string or list of strings specifying a volume(s) you wish your jobs to have access to.

cloud.join(self, jids, timeout=None, ignore_errors=False, deadlock_check=True)

Block current thread of execution until the job specified by the integer jids completes. Completion is defined as the job finishing or erroring (including stalling). If the job errored, a CloudException detailing the exception that triggered the error is thrown. If the job does not exist, a CloudException is thrown.

This method also accepts an iterable describing jids and blocks until all corresponding jobs finish. If an error is seen, join may terminate before all jobs finish. If multiple errors occur, it is undefined which job’s exception will be raised.

If timeout is set to a number, join will raise a CloudTimeoutError if the job is still running after timeout seconds

If ignore_errors is True, no CloudException will be thrown if a job errored. Join will block until every job is complete, regardless of error status.

If deadlock_check is True (default), join will error if it believes deadlock may have occured, as described in our docs

cloud.status(self, jids)

Returns the status of the job specified by the integer jids. If the job does not exist, a CloudException is thrown. This method also accepts an iterable describing jids, in which case a respective list of statuses is returned.

cloud.result(self, jids, timeout=None, ignore_errors=False, deadlock_check=True)

Blocks until the job specified by the integer jids has completed and then returns the return value of the job. If the job errored, a CloudException detailing the exception that triggered the error is thrown. If the job does not exist, a CloudException is thrown.

This function also accepts an iterable describing jids, in which case a respective list of return values is returned.

If timeout is set to a number, result will raise a CloudTimeoutError if the job is still running after timeout seconds

If ignore_errors is True, a job that errored will not raise an exception. Instead, its return value will be the CloudException describing the error.

If deadlock_check is True (default), result will error if it believes deadlock may have occured, as described in our docs

cloud.map(self, func, *args, **kwargs)

Map func (a callable) over argument sequence(s). cloud.map is meant to mimic a regular map call such as:

map(lambda x,y: x*y, xlist, ylist)

args can be any number of iterables. If the iterables are of different size, None is substituted for a missing entries. Note thatunlike call, map does not support passing kwargs into func

Map will return an iterable describing integer Job IDentifiers (jids). Each jid corresponds to func being invoked on one item of the sequence(s) (a ‘job’). In practice, the jids can (and should) be treated as a single jid; the returned iterable may be passed directly into status, result, etc.

Using cloud.result on the returned jids will return a list of job return values (each being the result of applying the function to an item of the argument sequence(s)).

Example:

def mult(x,y):
    return x*y
jids = cloud.map(mult, [1,3,5], [2,4,6]) 

This will cause mult3 to be invoked on PiCloud’s cluster with x=2

Results:

cloud.result(jids)
>> [2,12,30]

The result is [1*2,3*4,5*6]

See online documentation for additional information.

..note:: If you wish to have a pass a constant value across all map jobs, recommended practice is

to use closures to close the constant. The Lambda keyword is especially elegant:

cloud.map(lambda a, c: foo(a=a, b=b_constant, c=c), a_list, c_list)

Reserved special kwargs (see docs for details) are very similar to call:

  • _cores:

    Set number of cores your jobs will utilize. See http://docs.picloud.com/primer.html#choose-a-core-type/ In addition to having access to more CPU cores, the amount of RAM available will grow linearly. Possible values for _cores depend on what _type you choose:

    • c1: 1
    • c2: 1, 2, 4, 8
    • f2: 1, 2, 4, 8, 16
    • m1: 1, 2, 4, 8
    • s1: 1
  • _depends_on:

    An iterable of jids that represents all jobs that must complete successfully before the jobs created by this map function may be run.

  • _depends_on_errors:

    A string specifying how an error with a jid listed in _depends_on should be handled:

    • ‘abort’: Set this job to ‘stalled’ (Default)
    • ‘ignore’: Treat an error as satisfying the dependency
  • _env:

    A string specifying a custom environment you wish to run your jobs within. See environments overview at http://docs.picloud.com/environment.html

  • _fast_serialization:

    This keyword can be used to speed up serialization, at the cost of some functionality. This affects the serialization of both the arguments and return values Possible values keyword are:

    1. default – use cloud module’s enhanced serialization and debugging info
    2. no debug – Disable all debugging features for arguments
    3. use cPickle – Use Python’s fast serializer, possibly causing PicklingErrors

    func will always be serialized by the enhanced serializer (with debugging info).

  • _kill_process:

    Terminate the job’s Python interpreter after func completes, preventing the interpreter from being used by subsequent jobs. See Technical Overview for more info.

  • _label:

    A user-defined string label that is attached to the job. Labels can be used to filter when viewing jobs interactively (i.e. on the PiCloud website).

  • _max_runtime:

    Specify the maximum amount of time (in integer minutes) a job can run. If the job runs beyond this time, it will be killed.

  • _os_env_vars:

    List of operating system environment variables that should be copied to PiCloud from your system Alternatively a dictionary mapping the environment variables to the desired values.

  • _priority:

    A positive integer denoting the job’s priority. PiCloud tries to run jobs with lower priority numbers before jobs with higher priority numbers.

  • _profile:

    Set this to True to enable profiling of your code. Profiling information is valuable for debugging, and finding bottlenecks. Warning: Profiling can slow your job significantly; 10x slowdowns are known to occur

  • _restartable:

    In the rare event of hardware failure while the job is processing, this flag indicates whether the job can be restarted. By default, this is True. Jobs that modify external state (e.g. adding rows to a database) should set this False to avoid potentially duplicate modifications to external state.

  • _type:

    Choose the type of core to use, specified as a string:

    • c1: 1 compute unit, 300 MB ram, low I/O (default)
    • c2: 2.5 compute units, 800 MB ram, medium I/O
    • f2: 5.5 compute units, 3.75 GB ram, high I/O, hyperthreaded core
    • m1: 3.25 compute units, 8 GB ram, high I/O
    • s1: Up to 2 compute units (variable), 300 MB ram, low I/O, 1 IP per core

    See http://www.picloud.com/pricing/ for pricing information

  • _vol:

    A string or list of strings specifying a volume(s) you wish your job to have access to.

cloud.iresult(self, jids, timeout=None, num_in_parallel=10, ignore_errors=False)

Similar to result, but returns an iterator that iterates, in order, through the return values of jids as the respective job finishes, allowing quicker access to results and reducing memory requirements of result reading.

If a job being iterated over errors, an exception is raised. However, the iterator does not exhaust; if the exception is caught, one can continue iterating over the remaining jobs.

If timeout is set to a number, a call to the iterator’s next function will raise a CloudTimeoutError after timeout seconds if no result becomes available.

num_in_parallel controls how many results are read-ahead from the cloud Set this to 0 to use the allowed maximum.

If ignore_errors is True, a job that errored will return the CloudException describing its error, rather than raising an Exception

cloud.info(self, jids, info_requested=None)

Request information about jobs specified by integer or iterable jids

As this function is designed for console debugging, the return value is a dictionary whose keys are the jids requested. The values are themselves dictionaries, whose keys in turn consist of valid values within the iterable info_requested. Each key maps to key-specific information about the job, e.g. stdout maps to standard output of job.

Possible info_requested items are one of more of:

  • status: Job’s status
  • stdout: Standard Outout produced by job (last 64k characters)
  • stderr: Standard Error produced by job (last 64k characters)
  • logging: Logs produced with the Python logging module.
  • pilog: Logs generated by PiCloud regarding your job.
  • exception: Exception raised (on error)
  • runtime: runtime (wall clock) of job, specifically how long function took to evaluate
  • created: datetime when job was created on picloud by a call/map
  • finished: datetime when job completed running. note that finished - created = runtime + time in queue
  • env: Environment job runs under
  • vol: list of volumes job runs under
  • function: Name of function
  • label: Label of function
  • args: Arguments to function
  • kwargs: Keywork arguments to function
  • func_body: Actual function object
  • attributes: All attributes of job that were defined at its creation (including env and vol)
  • profile: Profiling information of the function that ran (if _profile was True)
  • memory: Variety of information about memory usage, swap usage, disk usage, and allocation failures
  • cputime: Amount of time job has spent consuming the cpu (in user and kernel mode)
  • cpu_percentage: Percent of cpu (core) job is currently occupying (in user and kernel mode) (only available as job is processing)
  • ports: Ports this job has opened (only available as job is processing).

If info_requested is None, info_requested will be status, stdout, stderr, and runtime. Set info_requested to ‘all’ to obtain info about every possible item

e.g. cloud.info(42,’stdout’) to get standard output of job 42
cloud.kill(self, jids=None)

Kill any incomplete jobs specified by the integer or iterable jids. If no arguments are specified (or jids is None), kill every job that has been submitted.

cloud.delete(self, jids)

Remove all data (result, stdout, etc.) related to jobs specified by the integer or iterable jids Jobs must have finished already to be deleted.

Note

In MP/Simulator mode, this does not delete any datalogs, only in-memory info

cloud.connection_info(self)

Returns a dictionary of information describing the current connection.

cloud.is_simulated(self)

Returns true iff cloud processor is being run on the local machine.

cloud.running_on_cloud(cls)

Returns true iff current thread of execution is running on PiCloud servers.

exception cloud.CloudException

Represents an exception that has occurred by a job on the cloud. CloudException will display the job id for failed cloud requests.

exception cloud.CloudTimeoutError

CloudException specifically for join timeouts

cloud.setkey(api_key, api_secretkey=None, server_url=None, restart=False, immutable=False)

Connect cloud to your PiCloud account, given your api_key. Your api_key is provided by PiCloud on the API Keys section of the PiCloud website.

The api_secretkey is generally stored on your machine. However, if you have not previously used this api_key or selected it in ‘picloud setup’, you will need to provide it.

server_url specifies the PiCloud server to connect to. Leave this blank to auto-resolve servers.

restart forces the cloud to reconnect

This command will disable the simulator if it is running.

cloud.start_simulator(force_restart=False)

Simulate cloud functionality locally. If force_restart or not already in simulation mode, restart cloud and enter simulation. In simulation mode, the cloud will be run locally, on multiple processors, via the multiprocessing library. Additional logging information will be enabled. For more information, see PiCloud documentation

cloud.set_dependency_whitelist(whitelist)

By default all relevant dependencies found will be transported to PiCloud In secure scenarios, you may wish to restrict the dependencies that may be transferred.

whitelist should be a list consisting of module names that can be transported.

Example:

['mypackage','package2.mymodule'] 

* Allows mypackage, package2.mymodule
* Disallows foo_module, package2.mymodule2 

Set to None to re-enable full automatic dependency magic

Warning

Dependency whitelist will reset if setkey() or start_simulator() is run

cloud.getconfigpath()

Return the directory where PiCloud configuration and logging files are stored. Configuration/logs are stored on a per-user basis

cloud.close(self)

Terminate cloud connection (or simulation) and all threads utilized by cloud. Returns True if closed cloud; False if cloud was already closed

Note

cloud will automatically re-open if you invoke any cloud.* again

Warning

This function should never be called by a job running on PiCloud

cloud.mp — Multiprocessing Interface

Cloud Multiprocessing Interface

cloud.mp has effectively the same interface as cloud.

All jobs will be run locally, on multiple processors, via the multiprocessing library

Sample usage:

import cloud.mp
jid = cloud.mp.call(lambda: 3*3)
>> Returns a job identifier
cloud.mp.result(jid)
>> Returns 9

cloud.bucket — PiCloud Object Storage Interface

Buckets is a key-value interface to store and retrieve data objects up to 5 GB in size.

A full overview of the bucket interface may be found within PiCloud’s documentation.

This module provides a python interface to your bucket.

In general, the various functions of this module use one or more of the following parameters:

  • obj_path: the key (path) of objects stored within the PiCloud bucket data store

  • prefix: Easy way to namespace objects; If present, the effective_obj_path is prefix + / + obj_path.

    If not present, the effective_obj_path is just obj_path. Referring to an object by obj_path=”file”, prefix=”folder” is the same as obj_path=”folder/file”, prefix=”“

  • file_path: Refers to a path on the local file system.

For convenience, in “copy” type functions (get, put, sync_to_cloud, etc.), the “destination” may be left blank. If so, the destination will be set to os.path.basename(source). (e.g. to upload, you need not put(file_path=’foo’, obj_path=’foo’). put(file_path=’foo’) will suffice.)

Note

Objects appearing in the bucket data store do not automatically appear on a job’s mounted file system. Jobs must access bucket data with the get function.

cloud.bucket.put(file_path, obj_path=None, prefix=None)

Transfer the file located on the local filesystem at file_path to the PiCloud bucket.

The name of the uploaded object will be the effective_obj_path (see module help) If obj_path is omitted, the name will be the basename of file_path.

The uploaded object can be retrieved later with the get function.

Example:

cloud.bucket.put('data/names.txt') 

This will upload data/names.txt from the local filesystem to the PiCloud bucket and store the object with the effective_obj_path names.txt. It can be later retrieved via cloud.bucket.get(‘names.txt’)

cloud.bucket.sync_to_cloud(file_path, obj_path=None, prefix=None)

Update bucket object if it has changed.

put(file_path, obj_path, prefix) only if contents of file_path on local file system differ from bucket object effective_obj_path (or bucket object does not exist)

cloud.bucket.putf(f, obj_path, prefix=None)

Similar to put, but accepts any file-like object (file, StringIO, etc.) f in lieu of a file_path. The contents of the uploaded object will be f.read()

The file-like object f must be seekable. Note that after putf returns, f will be closed

cloud.bucket.get(obj_path, file_path=None, prefix=None, start_byte=0, end_byte=None, _retries=1)

Retrieve the bucket object referenced by effective_obj_path from PiCloud and save it to the local file system at file_path.

Example:

cloud.bucket.get('names.txt','data/names.txt') 

This will retrieve the names.txt bucket object from PiCloud and save it locally to data/names.txt.

If file_path is None, it will be set to the basename of obj_path.

An optional byte_range can be specified using start_byte and end_byte, where only the data between start_byte and end_byte is returned.

An end_byte of None or exceeding file size is interpreted as a request to retrieve to end of file.

cloud.bucket.sync_from_cloud(obj_path, file_path=None, prefix=None)

Download bucket object if it has changed.

get(obj_path, file_path, prefix) only if contents of file_path on local file system differ from bucket object effective_obj_path (or local file does not exist)

cloud.bucket.getf(obj_path, prefix=None, start_byte=0, end_byte=None)

Retrieve the object referenced by effective_obj_path from PiCloud. Return value is a CloudBucketObject (file-like object) that can be read() to retrieve the object’s contents

An optional byte_range can be specified using start_byte and end_byte, where only the data between start_byte and end_byte is returned and made accessible to the CloudBucketObject. The returned CloudBucketObject.tell() will initialized to start_byte.

An end_byte of None or exceeding file size is interpreted as a request to retrieve to end of file.

class cloud.bucket.CloudBucketObject(action, ticket, file_size, start_byte=0, end_byte=None)

A CloudBucketObject provides a file-like interface to the contents of an object Seeking is not available.

close()

Close the object, blocking further I/O operations

md5

The md5 checksum of the contents Return None if not available

read(size=-1)

read([size]) -> read at most size bytes, returned as a string.

If the size argument is negative or omitted, read until EOF is reached.

readline(size=-1)

readline([size]) -> next line from the file, as a string.

Retain newline. A non-negative size argument limits the maximum number of bytes to return (an incomplete line may be returned then). Return an empty string at EOF.

readlines(sizehint=0)

readlines([size]) -> list of strings, each a line from the file.

Call readline() repeatedly and return a list of the lines so read. The optional size argument, if given, is an approximate bound on the total number of bytes in the lines returned.

tell()

Returns current file position as an integer

cloud.bucket.exists(obj_path, prefix=None)

Return boolean indicating if PiCloud bucket object named effective_obj_path exists

cloud.bucket.info(obj_path, prefix=None)

Return information about the PiCloud bucket object effective_obj_path

Information includes size, created time, last modified time, md5sum, public URL (if any), and any headers set with make_public

cloud.bucket.get_md5(obj_path, prefix=None)

Return the md5 checksum of the PiCloud bucket object effective_obj_path Return None if not available

cloud.bucket.iterlist(prefix=None, folderize=False, num_readahead_keys=1000)

Retrieve obj_paths of all objects stored on PiCloud. Returns an iterator that produces keys in lexicographic order. Unlike list() this guarantees that all keys will be returned:

  • prefix: Return only keys beginning with prefix.

  • folderize: Treat listing as directory based; compact keys containing “/” into a single folder

    (a key is a folder if and only if it ends in “/”) A folder can then be inspected by setting prefix equal to the folder name

  • num_readahead_keys: Number of keys to prefetch from amazon

Warning

Usage of iterlist is not recommended if result set will exceed 3,000 entries. Consider using prefixes and folderize to narrow results

cloud.bucket.list(prefix=None, folderize=False, marker=None, max_keys=1000)

Retrieve obj_paths of all objects stored on PiCloud. Returns a list of keys in lexicographic order.

  • prefix: Return only keys beginning with prefix.

  • folderize: Treat listing as directory based; compact keys containing “/” into a single folder

    (a key is a folder if and only if it ends in “/”) A folder can then be inspected by setting prefix equal to the folder name

  • marker: Return only keys with where key > marker

  • max_keys: Maximum number of keys that can be returned (max 1000). Fewer may be returned

The list will have an attribute, truncated, that indicates if the listing is truncated. To see the next results, make a subsequent list query with marker set to list[-1]

Use iterlist to avoid truncation

cloud.bucket.remove(obj_paths, prefix=None)

Removes object(s) named effective_obj_paths from PiCloud bucket

obj_paths can be a single object or a list of objects

cloud.bucket.remove_prefix(prefix)

Removes all objects beginning with prefix

cloud.bucket.make_public(obj_path, prefix=None, headers={}, reset_headers=False)

Makes the PiCloud bucket object effective_obj_path publicly accessible by a URL Returns public URL

Additionally, you can control the HTTP headers that will be in the response to a request for the URL with the headers dictionary.

Possible standard HTTP headers are:

  • content-type
  • content-encoding
  • content-disposition
  • cache-control

All other headers are considered custom and will have x-amz-meta- prepended to them.

Example: make_public(‘foo’,headers={‘content-type’: ‘text/x-python’, ‘purpose’ : ‘basic_script’} might return https://s3.amazonaws.com/pi-user-buckets/ddasy/foo

The headers in the response to a request for https://s3.amazonaws.com/pi-user-buckets/ddasy/foo will include:

  • content-type: text/x-python
  • x-amz-meta-purpose: basic_script

Clear all custom headers, other than content-type and content-encoding, by setting reset_headers to True

Note

Default content-type and content-encoding are inferred during the original cloud.bucket.put(..) call from the file_path and obj_path.

cloud.bucket.is_public(obj_path, prefix=None)

Determine if the PiCloud bucket object effective_obj_path is publicly accessible by a URL

Return public URL if it is; otherwise False

cloud.bucket.make_private(obj_path, prefix=None)

Removes the public URL associated with the PiCloud bucket object effective_obj_path

cloud.bucket.public_url_folder()

Return HTTP path that begins all your public bucket URLs. e.g. object ‘foo’ (if is_public) will be found at

public_url_folder() + foo

cloud.volume — PiCloud Volume Interface

PiCloud volume management. This module allows the user to manage their volumes on PiCloud. See documentation at http://docs.picloud.com

Be advised that you must have rsync installed for this module to work

cloud.volume.get_list(name=None, desc=False)

Returns a list of dictionaries describing user’s volumes. If name is specified, only shows info for the volume with that name. If desc is True (default=False), then the description is also displayed.

Volume information is returned as list of dictionaries. The keys within each returned dictionary are:

  • name:

    name of the volume

  • desc:

    description of the volume (if desc option is True)

  • mnt_path:

    filesystem path where volume contents can be accessed by a job

  • created:

    time when the volume was created

cloud.volume.create(name, mount_path, desc=None)

Creates a new cloud volume. * name:

name of the new volume (max 64 chars)
  • mount_path:

    If an absolute path is specified, that path is where this volume will be mounted when jobs are run specifying access to this volume, i.e. mount point where jobs can access the contents of this volume. If a relative path is specified, then the mount point is the specified path relative to /home/picloud, which is the directory where all jobs initially start.

  • desc:

    (optional) description of the volume (max 1024 chars)

cloud.volume.delete(name)

Deletes the picloud volume identified by name.

cloud.volume.sync(source, dest, delete=False)

Syncs data between a cloud volumes and the local filesystem.

Either source or dest should specify a cloud volume path, but not both. A cloud volume path is of the format:

volume_name:[path-within-volume]

where path-within-volume cannot be an absolute path (There is no concept of the root of the filesystem in a volume: All path specifications are relative to the top level of the volume). Note that the colon is what indicates this is a volume path specification. Local paths should point to a local directory or file. If the local path is a directory, whether the directory itself or the contents of the directory are synced depends on the presence of a trailing slash. A trailing slash indicates that the contents should be synced, while its absence would lead to the directory itself being synced to the volume. source can be a list of paths, all of which should either be local paths, or volume paths in the same cloud volume.

Example:

sync('~/dataset1', 'myvolume1:')

will ensure that a directory named ‘dataset1’ will exist at the top level of the cloud volume ‘myvolume1’, that contains all the contents of ‘dataset1’. On the other hand,

sync(‘~/dataset1/’, ‘myvolume1:’)

will copy all the contents of ‘dataset1’ to the top level of ‘myvolume1’. This behavior mirrors the file-copying tool ‘rsync’.

If delete is True, files that exist in dest but not in source will be deleted. By default, such files will not be removed.

cloud.volume.ls(volume_path, extended_info=False)

Lists the contents at volume_path.

  • volume_path:

    A cloud volume path spec or a list of specs, whose contents are to be returned.

  • extended_info:

    If True, in addition to the names of files and directories comprising the contents of the volume_path, the size (in bytes) and the modified times are returned. (Default is False)

Returns a list of tuples, one for each volume path specified. The first element of the tuple is the volume path spec, and the second element of the tuple is a list of dictionaries for each file or directory present in the volume path.

cloud.volume.rm(volume_path, recursive=False)

Removes contents at volume_path.

  • volume_path:

    A cloud volume path spec or a list of specs, whose contents are to be removed.

  • recursive:

    If True, will remove the contents at volume_path recursively, if it is a directory. If recursive is False, and volume_path points to a non-empty directory, it is an error. (Default is False)

cloud.queue — PiCloud Queue Interface

Queues provide an interface for flow-based programming on PiCloud.

Please see our Queue documentation.

cloud.queue.get(name)

Returns a CloudQueue object with name.

Note that returned CloudQueue is lazily registered on PiCloud. Until an operation with the queue is executed, exists(name) will return False, and the queue will not appear in list()

cloud.queue.list()

Returns a list of CloudQueue objects representing all registered Queues.

Registration is lazy. See get() for more info.

cloud.queue.exists(name)

Returns True if a CloudQueue of name is registered.

class cloud.queue.CloudQueue(name)

A CloudQueue is a queue registered on PiCloud.

All messages in the queue are stored on PiCloud. Any machine can pop these messages, or push more, by creating a CloudQueue with the same name.

ack(message_tickets)

Ack(nowledge) the ticket - and successful handling of - multiple message_tickets. If acknowledge is not called, the message will be automatically placed back into the queue when deadline is reached.

message_tickets can be a MessageTicket or the actual ticket id.

attach(message_handler, output_queues=[], iter_output=False, readers_per_job=1, max_parallel_jobs=1, retry_on=[], max_retries=None, retry_delay=None, on_error={}, **kwargs)

Register message_handler as the processor for this queue on PiCloud.

If message_handler is a function (or any callable), then it will be invoked for each message, with the message as the only argument.

The queue processing will occur in the context of a job. The job automatically ends if no data is available in the queue for 20 seconds.

Alternatively, message_handler may be an instance if it is desired to maintain state between message processing. In this case, the mssage_handler function will be the instance’s method “message_handler()” which takes a single argument (a message). The instance may optionally provide pre_handling and post_handling methods to manage startup and cleanup respectively. pre_handling is called once before the first message is processed by the wrapping job, and post_handling is called once after the last message is processed by the wrapping job.

By default, the return value of the message_handler is pushed to each queue in output_queues. If the return value is None, no message is pushed.

If iter_output is set to True, the return value of the message_handler will be considered an iterable. Each element of the iterator (including any None values) will be pushed to each queue in output_queues.

readers_per_job controls the number of messages can be processed by a job in parallel.

max_parallel_jobs is the maximum number of jobs that can run in parallel to process your queue.
Set to 0 to disable processing

retry_on, max_retries, retry_delay and on_error all relate to error handling. Please see our online documentation: http://docs.picloud.com/queue.html#error-handling

Certain special kwargs associated with cloud.call can be used for your message_handler:

  • _cores:

    Set number of cores your job will utilize. See http://docs.picloud.com/primer.html#choose-a-core-type/ In addition to having access to more CPU cores, the amount of RAM available will grow linearly. Possible values for _cores depend on what _type you choose:

    • c1: 1
    • c2: 1, 2, 4, 8
    • f2: 1, 2, 4, 8, 16
    • m1: 1, 2, 4, 8
    • s1: 1
  • _env:

    A string specifying a custom environment you wish to run your job within. See environments overview at http://docs.picloud.com/environment.html

  • _label:

    A user-defined string label that is attached to the jobs. Labels can be used to filter when viewing jobs interactively (i.e. on the PiCloud website).

  • _max_runtime:

    Specify the maximum amount of time (in integer minutes) jobs can run. If the job runs beyond this time, it will be killed (messages will be rerun in a new job)

  • _os_env_vars:

    List of operating system environment variables that should be copied to PiCloud from your system Alternatively a dictionary mapping the environment variables to the desired values.

  • _priority:

    A positive integer denoting the jobs’ priority. PiCloud tries to run jobs with lower priority numbers before jobs with higher priority numbers.

  • _profile:

    Set this to True to enable profiling of your code. Profiling information is valuable for debugging, and finding bottlenecks. Warning: Profiling can slow your job significantly; 10x slowdowns are known to occur

  • _restartable:

    In the rare event of hardware failure while a job is processing, this flag indicates whether the job can be restarted. By default, this is True. Jobs that modify external state (e.g. adding rows to a database) should set this False to avoid potentially duplicate modifications to external state.

  • _type:

    Choose the type of core to use, specified as a string:

    • c1: 1 compute unit, 300 MB ram, low I/O (default)
    • c2: 2.5 compute units, 800 MB ram, medium I/O
    • f2: 5.5 compute units, 3.75 GB ram, high I/O, hyperthreaded core
    • m1: 3.25 compute units, 8 GB ram, high I/O
    • s1: Up to 2 compute units (variable), 300 MB ram, low I/O, 1 IP per core

    See http://www.picloud.com/pricing/ for pricing information

  • _vol:

    A string or list of strings specifying a volume(s) you wish your jobs to have access to.

count()

Returns an approximation of the number of messages in the queue.

delete()

Delete the queue. All messages will be lost.

detach()

Detaches any attached message handler

info()

Returns a dictionary describing the queue (notably attachment information)

pop(max_count=10, timeout=20)

Pops at most max_count messages from the queue. Returns a list of popped messages. Note that the non-FIFO order of the pop is intentional.

Blocks until at least one message is available, or until timeout seconds elapses, at which point an empty list is returned.

timeout must be specified as an integer between 0 and 20 inclusive. max_count must be specified as an integer between 1 and 10 inclusive.

pop_tickets(max_count=10, timeout=20, deadline=30)

Pops at most max_count messages from the queue. Returns a list of MessageTickets.

Semantics are similar to pop, except that instances of MessageTicket are returned which encapsulates the actual message.

All MessageTickets must be acknowledged within deadline seconds. If they are not the message will be re-added to the queue.

Using pop_tickets and MessageTicket.ack() (or ack_messages) allows you to build highly robust messaging systems where a message will go unprocessed.

push(messages, delay=0)

Put messages, a list of objects, into queue. If delay is specified, the function will return immediately, but the messages will not be available to readers (poppers) for delay seconds.

Each object in messages must be pickle-able.

Note that if the picked message exceeds 64kb, it will be temporarily saved to your cloud.bucket under the queue/ prefix.

update_deadline(message_tickets, new_deadline)

Modify the deadline of multiple not-yet ack-ed message_tickets After invoking this function, the message_tickets’ deadline will occur in new_deadline seconds relative to NOW.

message_tickets can be a message_ticket or the ticket id.

class cloud.queue.MessageTicket(queue, message, ticket, deadline, creation_time=None, redirect_key=None)

A MessageTicket includes information and functions for acknowleding a message, or changing its deadline. It also includes the message itself.

To access the message, read the message attribute. To access ticket id, read the ticket_id attribute

ack()

Acknowledge this ticket. For batch acknowledgements, see CloudQueue.acknowledge_message

time_until_deadline()

Returns how much time (in seconds) is left until the message’s deadline.

update_deadline(new_deadline)

If the deadline you initially set it insufficient (the message cannot be reliably purged from the queue), update its deadline. The new_deadline should be the number of seconds in the future (relative to now) before the message is re-added to the queue automatically.

exception cloud.queue.Retry

Raise this Exception within an attachment to abort and retry the message

Within the constructor, the delay muyst be set to specify how many seconds
to wait until the retried message can be dequeued. max_retries must also be set to specify how many times the message can be retried
exception cloud.queue.MaxRetryException

Signals maximum retries exceeded

cloud.cron — PiCloud Cron Interface

For managing crons on PiCloud. Crons allow you to “register” a function to be invoked periodically on PiCloud according to a schedule you specify.

Api keys must be configured before using any functions in this module. (via cloudconf.py, cloud.setkey, or being on PiCloud server)

cloud.cron.register(func, label, schedule, **kwargs)

Register func (a callable) to be run periodically on PiCloud according to schedule

The cron can be managed in the future by the specified label

Schedule is a BSD-style cron timestamp, specified as either a string of 5 non- whitespace characters delimited by spaces or as a sequence (tuple, list, etc.) of 5 elements, each with non-whitespace characters. The ordering of the characters represent scheduling information for minutes, hours, days, months, and days of week. See full format specification at http://unixhelp.ed.ac.uk/CGI/man-cgi?crontab+5 Year is not supported. The N/S increment format is though.

PiCloud schedules your crons under the GMT (UTC+0) timezone.

Certain special kwargs associated with cloud.call can be attached to the periodic jobs:

  • _cores:

    Set number of cores your job will utilize. See http://docs.picloud.com/primer.html#choose-a-core-type/ In addition to having access to more CPU cores, the amount of RAM available will grow linearly. Possible values for _cores depend on what _type you choose:

    • c1: 1
    • c2: 1, 2, 4, 8
    • f2: 1, 2, 4, 8, 16
    • m1: 1, 2, 4, 8
    • s1: 1
  • _env:

    A string specifying a custom environment you wish to run your job within. See environments overview at http://docs.picloud.com/environment.html

  • _fast_serialization:

    This keyword can be used to speed up serialization, at the cost of some functionality. This affects the serialization of the spawned jobs’ return value. The stored function will always be serialized by the enhanced serializer, with debugging features. Possible values keyword are:

    1. default – use cloud module’s enhanced serialization and debugging info
    2. no debug – Disable all debugging features for result
    3. use cPickle – Use python’s fast serializer, possibly causing PicklingErrors
  • _max_runtime:

    Specify the maximum amount of time (in integer minutes) jobs can run. If the job runs beyond this time, it will be killed.

  • _os_env_vars:

    List of operating system environment variables that should be copied to PiCloud from your system Alternatively a dictionary mapping the environment variables to the desired values.

  • _priority:

    A positive integer denoting the job’s priority. PiCloud tries to run jobs with lower priority numbers before jobs with higher priority numbers.

  • _profile:

    Set this to True to enable profiling of your code. Profiling information is valuable for debugging, but may slow down your jobs.

  • _restartable:

    In the very rare event of hardware failure, this flag indicates that a spawned job can be restarted if the failure happened in the middle of it. By default, this is true. This should be unset if the function has external state (e.g. it modifies a database entry)

  • _type:

    Choose the type of core to use, specified as a string:

    • c1: 1 compute unit, 300 MB ram, low I/O (default)
    • c2: 2.5 compute units, 800 MB ram, medium I/O
    • f2: 5.5 compute units, 3.75 GB ram, high I/O, hyperthreaded core
    • m1: 3.25 compute units, 8 GB ram, high I/O
    • s1: Up to 2 compute units (variable), 300 MB ram, low I/O, 1 IP per core

    See http://www.picloud.com/pricing/ for pricing information

  • _vol:

    A string or list of strings specifying a volume(s) you wish your jobs to have access to.

Note

The callable must take no arguments. To schedule functions that take arguments, wrap it with a closure. e.g. func(x,y) –> lambda: func(x,y)

cloud.cron.deregister(label)

Deregister (delete) the cron specified by label

cloud.cron.manual_run(label)

Manually run the function of a cron specified by label (ignoring schedule) Returns the ‘jid’ of the job created by this run command

cloud.cron.list()

List labels of active crons registered on PiCloud

cloud.cron.info(label)

Return a dictionary of relevant info about cron specified by label

Info includes:

  • label: The same as was passed in
  • schedule: The schedule of this cron
  • last_run: Time this cron was last run
  • last_jid: Last run jid of this cron
  • created: When this cron was created
  • creator_host: Hostname of the machine that registered this cron
  • func_name: Name of function associated with cron

cloud.rest — PiCloud REST Registration Interface

Interface to publish functions to PiCloud, allowing them to be invoked via the REST API

Api keys must be configured before using any functions in this module. (via cloudconf.py, cloud.setkey, or being on PiCloud server)

cloud.rest.publish(func, label, out_encoding='json', **kwargs)

Publish func (a callable) to PiCloud so it can be invoked through the PiCloud REST API

The published function will be managed in the future by a unique (URL encoded) label.

out_encoding specifies the format that the return value should be in when retrieving the result via the REST API. Valid values are “json” for a JSON-encoded object and “raw”, where the return value must be an str (but can contain any characters).

The return value is the URL which can be HTTP POSTed to to invoke func. See http://docs.picloud.com/rest.html for information about PiCloud’s REST API

Certain special kwargs associated with cloud.call can be attached to the periodic jobs:

  • _cores:

    Set number of cores your job will utilize. See http://docs.picloud.com/primer.html#choose-a-core-type/ In addition to having access to more CPU cores, the amount of RAM available will grow linearly. Possible values for _cores depend on what _type you choose:

    • c1: 1
    • c2: 1, 2, 4, 8
    • f2: 1, 2, 4, 8, 16
    • m1: 1, 2, 4, 8
    • s1: 1
  • _env:

    A string specifying a custom environment you wish to run your job within. See environments overview at http://docs.picloud.com/environment.html

  • _fast_serialization:

    This keyword can be used to speed up serialization, at the cost of some functionality. This affects the serialization of the spawned jobs’ return value. The stored function will always be serialized by the enhanced serializer, with debugging features. Possible values keyword are:

    1. default – use cloud module’s enhanced serialization and debugging info
    2. no debug – Disable all debugging features for result
    3. use cPickle – Use python’s fast serializer, possibly causing PicklingErrors
  • _max_runtime:

    Specify the maximum amount of time (in integer minutes) jobs can run. If the job runs beyond this time, it will be killed.

  • _os_env_vars:

    List of operating system environment variables that should be copied to PiCloud from your system Alternatively a dictionary mapping the environment variables to the desired values.

  • _priority:

    A positive integer denoting the job’s priority. PiCloud tries to run jobs with lower priority numbers before jobs with higher priority numbers.

  • _profile:

    Set this to True to enable profiling of your code. Profiling information is valuable for debugging, but may slow down your jobs.

  • _restartable:

    In the very rare event of hardware failure, this flag indicates that a spawned job can be restarted if the failure happened in the middle of it. By default, this is true. This should be unset if the function has external state (e.g. it modifies a database entry)

  • _type:

    Choose the type of core to use, specified as a string:

    • c1: 1 compute unit, 300 MB ram, low I/O (default)
    • c2: 2.5 compute units, 800 MB ram, medium I/O
    • f2: 5.5 compute units, 3.75 GB ram, high I/O, hyperthreaded core
    • m1: 3.25 compute units, 8 GB ram, high I/O
    • s1: Up to 2 compute units (variable), 300 MB ram, low I/O, 1 IP per core

    See http://www.picloud.com/pricing/ for pricing information

  • _vol:

    A string or list of strings specifying a volume(s) you wish your jobs to have access to.

cloud.rest.remove(label)

Remove a published function from PiCloud

cloud.rest.list()

List labels of published functions

cloud.rest.info(label)

Retrieve information about a published function specified by label

cloud.config — Configuration Interface

class cloud.util.configmanager.ConfigSettings(confmanager, do_reload=False)

This object provides the ability to programmatically edit the cloud configuration (found in cloudconf.py). commit() must be called to update the cloud module with new settings - and restart all active clouds

api_key

your application’s api key provided by PiCloud

automatic_dependency_transfer

Set to False to disable automatic dependency transfer. This will not affect dependencies already sent to PiCloud.

cloud_result_cache_size

Amount (in kb) of job results to hold in local memory; set to 0 to disable caching. This option only applies when connected to PiCloud.

cloud_status_cache_size

Number of job statuses to hold in local memory; set to 0 to disable caching. This option only applies when connected to PiCloud.

commit()

Update cloud with new settings.

Warning

This will restart any active cloud instances, wiping mp/simulated jobs and setkey information

force_redirect_job_output

If set and cloud is in simulation mode, forces redirect_job_output (see multiprocessing section) to true

force_serialize_logging

If this is is set and cloud is running in simulation mode, serialize_logging (see logging section) will be turned on

http_close

Should every HTTP connection be closed after receiving a response?

log_filename

Filename where cloud log messages should be written. This path is relative to ~/.picloud/

log_level

logging level for cloud messages. This affects both messages saved to the cloud log file and sent through the python logging system. See http://docs.python.org/library/logging.html for more information

max_transmit_data

Maximum amount of data (in bytes) that can be sent to the PiCloud server per function or function argument. Value may be raised up to 16 MB

mp_cache_size_limit

Maxmimum number of jobs that cloudmp should store. Set to 0 for no limit. This option only applies to cloud.mp and the cloud simulator. PiCloud recommends that any limit stays above 1024.

num_procs

Number of subprocesses that cloud multiprocessing should utilize. Beware of using too low of a number of subprocesses, as deadlock can occur. If set to 0, this will be set to the number of cores available on this system.

print_log_level

logging level for printing cloud log messages to console. Must be equal or higher than log_level

proxy_server

Specify an http/https proxy server that should be used e.g. proxy.example.com:3128 for anonymous or username:password@proxy.example.com:3128 for authenticated

purge_days

Number of days to wait until purging old serialization log directories

redirect_job_output

If set to true, job stdout and stderr will be redirected to files inside the serialize_logging_path. This option simulates PiCloud behavior and must be set to true for cloud.info to be able to request stdout or stderr information about jobs. If set to false, job stdout and stderr will be the same as the parent process.

save_log

Should log_filename (default of cloud.log) be written with cloud log messages? Note that cloud will always emit logging messages; this option controls if cloud should have its own log.

serialize_logging

Should all object serialization meta-data be logged to the serialize_logging_path?

serialize_logging_path

Path to save object serialization meta-data. This path is relative to ~/.picloud/

server_list_url

url to list of PiCloud servers

use_gzip

Request gziped HTTP responses

use_simulator

The Cloud Simulator simulates cloud.* functions locally, allowing for quicker debugging. If this is enabled, all cloud.* functions will be run locally, rather than on PiCloud. See web documentation.

cloud.realtime — PiCloud Realtime Management Interface

PiCloud realtime management. This module allows the user to manage their realtime cores programmatically. See http://docs.picloud.com/tech_overview.html#scheduling

cloud.realtime.list(request_id='')

Returns a list of dictionaries describing realtime core requests. If request_id is specified, only show realtime core request with that request_id

The keys within each returned dictionary are:

  • request_id: numeric ID associated with the request
  • type: Type of computation resource this request grants
  • cores: Number of (type) cores this request grants
  • start_time: Time when real time request was satisfied; None if still pending
cloud.realtime.request(type, cores, max_duration=None)

Request a number of cores of a certain compute resource type Returns a dictionary describing the newly created realtime request, with the same format as the requests returned by list_rt_cores. If specified, request will terminate after being active for max_duration hours

cloud.realtime.release(request_id)

Release the realtime core request associated with request_id. Request must have been satisfied to terminate.

cloud.environment — PiCloud Environment Interface

PiCloud environment management. This module allows the user to manage their environments on PiCloud. See documentation at http://docs.picloud.com

cloud.environment.list_envs(name=None)

Returns a list of dictionaries describing user’s environments. If name is given, only shows info for the environment with that name.

Environment information is returned as list of dictionaries. The keys within each returned dictionary are:

  • name: name of the environment
  • status: status of the environment
  • action: the action state of the environment (e.g. under edit)
  • created: time when the environment was created
  • last_modifed: last time a modification was saved
  • hostname: hostname of setup server if being modified
  • setup_time: time setup server has been up if being modified
cloud.environment.list_bases()

Returns a list of dictionaries describing available bases. The keys within each returned dictionary are:

  • id: id of the base (to be used when referencing bases in other functions)
  • name: brief descriptive name of the base
cloud.environment.create(name, base, desc=None)

Creates a new cloud environment.

  • name: name of the new environment (max 30 chars)

  • base: name of the base OS to use for the environment (use list_bases to

    see list of bases and their names)

  • desc: Optional description of the environment (max 2000 chars)

Returns the hostname of the setup server where the newly created environment can be modified.

cloud.environment.edit_info(name, new_name=None, new_desc=None)

Edits name and description of an existing environment.

  • name: current name of the environment
  • new_name: Optional new name of the environment (max 30 chars)
  • new_desc: Optional new description of the environment (max 2000 chars)
cloud.environment.modify(name)

Modifies an existing environment.

  • name: name of environment to modify

Returns the hostname of the setup server where environment can be modified.

cloud.environment.save(name)

Saves the current modified version of the environment, without tearing down the setup server.

  • name: name of the environment to save

This is a blocking function. When it returns without errors, the new version of the environment is available for use by all workers.

cloud.environment.save_shutdown(name)

Saves the current modified version of the environment, and tears down the setup server when saving is done.

  • name: name of the environment to save

This is a blocking function. When it returns without errors, the new version of the environment is available for use by all workers.

cloud.environment.shutdown(name)

Tears down the setup server without saving the environment modification.

  • name: name of the environment to save
cloud.environment.clone(parent_name, new_name=None, new_desc=None)

Creates a new cloud environment by cloning an existing one.

  • parent_name: name of the existing environment to clone

  • new_name: new name of the environment. default is

    parent_name + “_clone”. (max 30 chars)

  • new_desc: Optional description of the environment if different from

    parent environment description. (max 2000 chars)

cloud.environment.get_setup_hostname(name)

Returns the hostname of the setup server where environment can be modified. raises exception if the environment does not have a setup server already launched.

  • name: name of the environment whose setup server hostname is desired
cloud.environment.get_key_path()

Return the key file path for sshing into setup server.

cloud.environment.ssh(name, cmd=None)

Creates an ssh session to the environment setup server.

  • name: Name of the environment to make an ssh connection

  • cmd: By default, this function creates an interactive ssh session.

    If cmd is given, however, it executes the cmd on the setup server and returns the output of the command execution.

cloud.environment.rsync(src_path, dest_path, delete=False, pipe_output=False, root=False)

Syncs data between a custom environment and the local filesystem. A setup server for the environment must already be launched. Also, keep in mind that the picloud user account (which is used for the rsync operation) has write permissions only to the home directory and /tmp on the setup server. If additional permissions are required, consider doing the rsync manually from the setup server using sudo, or rsync to the home directory then do a subsequent move using sudo.

Either src_path or dest_path should specify an environment path, but not both. An environment path is of the format:

env_name:[path-within-environment]

Note that the colon is what indicates this is an environment path specification.

src_path can be a list of paths, all of which should either be local paths, or environment paths. If src_path is a directory, a trailing slash indicates that its contents should be rsynced, while ommission of slash would lead to the directory itself being rsynced to the environment.

Example:

rsync('~/dataset1', 'my_env:')

will ensure that a directory named ‘dataset1’ will exist in the user picloud’s home directory of environment ‘my_env’. On the other hand,

rsync(~/dataset1/’, ‘my_env:’)

will copy all the contents of ‘dataset1’ to the home directory of user picloud. See rsync manual for more information.

If delete is True, files that exist in dest_path but not in src_path will be deleted. By default, such files will not be removed.

cloud.environment.run_script(name, filename)

Runs a script on the environment setup server, and returns the output.

  • name: Environment whose setup server should run the script

filename: local path where the script to be run can be found

cloud.account — Account Management Interface

PiCloud account management. This module allows the user to manage account information programmatically. Currently supports api key management

cloud.account.list_keys(username, password, active_only=False)

Returns a list of all api keys. If active_only is True, only active keys are returned. username and password should be your PiCloud login information.

cloud.account.get_key(username, password, api_key)

Returns information including api_secretkey, active status, and note for the specified api_key. username and password should be your PiCloud login information.

cloud.account.get_key_by_key(api_key, api_secretkey)

Similar to get_key, but access information via api_key credentials (api_key and api_secretkey).

cloud.account.activate_key(username, password, api_key)

Activates the specified api_key. username and password should be your PiCloud login information.

cloud.account.deactivate_key(username, password, api_key)

Deactivates the specified api_key. username and password should be your PiCloud login information.

cloud.account.create_key(username, password)

Creates a new api_key. username and password should be your PiCloud login information.

cloud.shell — Run shell commands on PiCloud

For running commands in a shell environment on PiCloud. Shell programs are executed through template substitution

cloud.shell.execute(command, argdict, return_file=None, ignore_exit_status=False, cwd=None, **kwargs)

Execute (possibly) templated command. Returns Job IDentifier (jid)

  • argdict - Dictionary mapping template parameters to values
  • return_file: Contents of this file will be result of job. result is stdout if not provided
  • ignore_exit_status: if true, a non-zero exit code will not result in job erroring
  • cwd: Current working directory to execute command within
  • kwargs: See cloud.call underscored keyword arguments
cloud.shell.execute_map(command, common_argdict, map_argdict, return_file=None, ignore_exit_status=False, cwd=None, **kwargs)
Execute templated command in parallel. Return list of Job Identifiers (jids). See cloud.map
for more information about mapping. Arguments to this are:
  • common_argdict - Dictionary mapping template parameters to values for ALL map jobs

  • map_argdict - Dictionary mapping template parameters to a list of values

    The nth mapjob will have its template parameter substituted by the nth value in the list Note that all elements of map_argdict.values() must have the same length; The number of mapjobs produced will be equal to that length

  • return_file: Contents of this file will be result of job. result is stdout if not provided

  • ignore_exit_status: if true, a non-zero exit code will not result in job erroring

  • cwd: Current working directory to execute command within

  • kwargs: See cloud.map underscored keyword arguments

cloud.shell.rest_publish(command, label, return_file=None, ignore_exit_status=False, **kwargs)

Publish shell command to PiCloud so it can be invoked through the PiCloud REST API The published function will be managed in the future by a unique (URL encoded) label. Returns url of published function. See cloud.rest.publish

See cloud.shell.execute for description other arguments See cloud.rest.publish for description of **kwargs

cloud.shell.cron_register(command, label, schedule, return_file=None, ignore_exit_status=False, **kwargs)

Register shell command to be run periodically on PiCloud according to schedule The cron can be managed in the future by the specified label.

Flags only relevant if you call cloud.result() on the cron job: return_file: Contents of this file will be result of job created by REST invoke.

result is stdout if not provided

ignore_exit_status: if true, a non-zero exit code will not result in job erroring

cloud.pool_interface — Multiprocessing Pool Wrapper

A module provided that emulates python multiprocessing’s Process Pool interface. This can be used to trivially replace multiprocessing Pool code.

Note that cloud.setkey must still be used before using this interface.

Not supported (mostly due to not making sense in the PiCloud context).

  • Pool initializer
  • Passing keyword arguments to function being applied
  • callback for map/map_async
  • (i)map chunksize – argument only controls cloud.iresult chunksize
  • iresult_undordered – this is just an ordered iresult
  • multiprocessing.TimeoutError – cloud.CloudTimeoutError is used instead
  • join, close and terminate operations

Warning

It is highly recommended that you do not use this module and instead use the cloud module directly. Much functionality that the cloud interface offers is missing here.

cloud.pool_interface.apply(func, args=())

Equivalent to Multiprocessing apply. keyword arguments are not supported

cloud.pool_interface.apply_async(func, args=(), callback=None)

Equivalent to Multiprocessing apply_async keyword arguments are not supported

callback is a list of functions that should be run on the callee’s computer once this job finishes successfully.
Each callback will be invoked with one argument - the jid of the complete job
cloud.pool_interface.map(func, iterable, chunksize=None)

Equivalent to Multiprocessing map chunksize is not used here

cloud.pool_interface.map_async(func, iterable, chunksize=None)

Equivalent to Multiprocessing map_async chunksize is not used here

cloud.pool_interface.imap(func, iterable, chunksize=None)

Equivalent to Multiprocessing imap chunksize is used only to control the cloud.iresult stage

cloud.pool_interface.imap_unordered(func, iterable, chunksize=None)

Same as imap

class cloud.pool_interface.AsyncResult(jid)

Result object that emulates multiprocessing.pool.AsyncResult

get(timeout=None)

Return result when it arrives. If timeout is not None and none arrives, raise multiprocessing.TimeoutError in timeout seconds

ready()

Returns true if the job finished (done or errored)

successful()

Returns true if job finished successfully. Asserts that job has finished

wait(timeout=None)

Wait until result is available or timeout seconds pass