API Documentation

cloud — PiCloud Interface

Interface for cloud - the clientside module for PiCloud

Sample usage:

import cloud
jid = cloud.call(lambda: 3*3)
>> Returns a job identifier
cloud.result(jid)
>> Returns 9

This will run the function lambda: 3*3 on PiCloud’s cluster, and return the result. Most functions, even user-defined ones, can be passed through cloud.call

For cloud to work, you must first do one the following:

  1. Have set api_key and api_secretkey in $HOME/.picloud/cloudconf.py
  2. Call cloud.setkey( api_key=your_api_key, api_secretkey=your_api_secretkey)
  3. Set use_simulator to true in cloudconf.py or run cloud.start_simulator() to use the cloud simulator, where cloud is run locally
cloud.setkey(api_key, api_secretkey=None, server_url=None, restart=False, immutable=False)

Connect cloud to your PiCloud account, given your api_key. Your api_key is provided by PiCloud on the API Keys section of the PiCloud website.

The api_secretkey is stored on your machine. However, if you have not previously used this api_key or selected it in ‘picloud setup’, you will need to provide it.

server_url specifies the PiCloud server to connect to. Leave this blank to have cloud auto-resolve servers.

restart forces the cloud to reconnect

This command will disable the simulator if it is running.

cloud.start_simulator(force_restart=False)

Simulate cloud functionality locally. If force_restart or not already in simulation mode, restart cloud and enter simulation. In simulation mode, the cloud will be run locally, on multiple processors, via the multiprocessing library. Additional logging information by default will be enabled. For more information, see PiCloud documention

cloud.call(self, func, *args, **kwargs)

Invoke func (a callable) in the cloud. When invoked, func will be invoked on PiCloud’s cluster with the passed args that follow it. The invoked function is known as a ‘job’. The return value of the invoked function is known as the ‘internal result’.

Call will return an integer Job IDentifier (jid) which can be passed into the status and result methods to obtain the status of the job and the internal result respectively.

Example:

def mult3(x):
    return 3*x
cloud.call(mult3, 2) 

This will cause mult3 to be invoked on PiCloud’s cluster with x=2

See online documentation for additional information.

Reserved special kwargs (see docs for details):

  • _callback:

    A list of functions that should be run on the callee’s computer once this job finishes successfully.

  • _callback_on_error:

    A list of functions that should be run on the callee’s computer if this job errors.

  • _depends_on:

    An iterable of jids that represents all jobs that must complete successfully before the job created by this call function may be run.

  • _env:

    A string specifying a custom environment you wish to run your job within. See environments overview at http://blog.picloud.com/2011/09/26/introducing-environments-run-anything-on-picloud/

  • _fast_serialization:

    This keyword can be used to speed up serialization, at the cost of some functionality. This affects the serialization of both the arguments and return values func will always be serialized by the enhanced serializer, with debugging features. Possible values keyword are:

    1. default – use cloud module’s enhanced serialization and debugging info
    2. no debug – Disable all debugging features for arguments
    3. use cPickle – Use Python’s fast serializer, possibly causing PicklingErrors
  • _kill_process:

    Terminate the Python interpreter func runs in after func completes, preventing the interpreter from being used by subsequent jobs. See Technical Overview for more info.

  • _label:

    A user-defined string label that is attached to the job. Labels can be used to filter when viewing jobs interactively (i.e. on the PiCloud website).

  • _priority:

    A positive integer denoting the job’s priority. PiCloud tries to run jobs with lower priority numbers before jobs with higher priority numbers.

  • _profile:

    Set this to True to enable profiling of your code. Profiling information is valuable for debugging, but may slow down your job.

  • _restartable:

    In the very rare event of hardware failure, this flag indicates that the job can be restarted if the failure happened in the middle of the job. By default, this is true. This should be unset if the job has external state (e.g. it modifies a database entry)

  • _type:

    Select the type of compute resources to use. PiCloud supports four types, specified as strings:

    ‘c1’

    1 compute unit, 300 MB ram, low I/O (default)

    ‘c2’

    2.5 compute units, 800 MB ram, medium I/O

    ‘m1’

    3.25 compute units, 8 GB ram, high I/O

    ‘s1’

    Up to 2 compute units (variable), 300 MB ram, low I/O, 1 IP per core

    See http://www.picloud.com/pricing/ for pricing information

cloud.join(self, jids, timeout=None, ignore_errors=False)

Block current thread of execution until the job specified by the integer jids completes. Completion is defined as the job finishing or erroring (including stalling). If the job errored, a CloudException detailing the exception that triggered the error is thrown. If multiple errors occur, it is undefined which job’s exception will be raised. If the job does not exist, a CloudException is thrown.

This method also accepts an iterable describing jids and blocks until all corresponding jobs finish. If an error is seen, join may terminate before all jobs finish.

If timeout is set to a number, join will raise a CloudTimeoutError if the job is still running after timeout seconds

If ignore_errors is True, no CloudException will be thrown if a job errored. Join will block until every job is complete, regardless of error status.

cloud.status(self, jids)

Returns the status of the job specified by the integer jids. If the job does not exist, a CloudException is thrown. This method also accepts an iterable describing jids, in which case a respective list of statuses is returned.

cloud.result(self, jids, timeout=None, ignore_errors=False)

Blocks until the job specified by the integer jids has completed and then returns the internal result of the job. If the job errored, a CloudException detailing the exception that triggered the error is thrown. If the job does not exist, a CloudException is thrown.

This function also accepts an iterable describing jids, in which case a respective list of internal results is returned

If timeout is set to a number, result will raise a CloudTimeoutError if the job is still running after timeout seconds

If ignore_errors is True, a job that errored will not raise an exception. Instead, its return value is the CloudException describing the error.

cloud.map(self, func, *args, **kwargs)

Map func (a callable) over argument sequence(s). cloud.map is meant to mimic a regular map call such as:

map(lambda x,y: x*y, xlist, ylist)

args can be any number of iterables. If the iterables are of different size, ‘None’ is substituted for a missing entries.

Map will return an iterable describing integer Job IDentifiers (jids). Each jid corresponds to func being invoked on one item of the sequence(s) (a ‘job’). In practice, the jids can (and should) be treated as a single jid; the returned iterable may be passed directly into status, result, etc.

Using cloud.result on the returned jids will return a list of internal results (each being the result of applying the function to an item of the argument sequence(s)).

Example:

def mult(x,y):
    return x*y
jids = cloud.map(mult, [1,3,5], [2,4,6]) 

This will cause mult3 to be invoked on PiCloud’s cluster with x=2

Results:

cloud.result(jids)
>> [2,12,30]

The result is [1*2,3*4,5*6]

See online documentation for additional information

Reserved special kwargs (see docs for details):

  • _depends_on:

    An iterable of jids that represents all jobs that must complete successfully before any jobs created by this map function may be run.

  • _env:

    A string specifying a custom environment you wish to run your jobs within. See environments overview at http://blog.picloud.com/2011/09/26/introducing-environments-run-anything-on-picloud/

  • _fast_serialization:

    This keyword can be used to speed up serialization, at the cost of some functionality. This affects the serialization of both the map arguments and return values The map function will always be serialized by the enhanced serializer, with debugging features. Possible values keyword are:

    1. default – use cloud module’s enhanced serialization and debugging info
    2. no debug – Disable all debugging features for arguments
    3. use cPickle – Use python’s fast serializer, possibly causing PicklingErrors
  • _kill_process:

    Terminate the Python interpreter func runs in after func completes, preventing the interpreter from being used by subsequent jobs. See Technical Overview for more info.

  • _label:

    A user-defined string label that is attached to the created jobs. Labels can be used to filter when viewing jobs interactively (i.e. on the PiCloud website).

  • _priority:

    A positive integer denoting the job’s priority. PiCloud tries to run jobs with lower priority numbers before jobs with higher priority numbers.

  • _profile:

    Set this to True to enable profiling of your code. Profiling information is valuable for debugging, but may slow down your job.

  • _restartable:

    In the very rare event of hardware failure, this flag indicates that the job can be restarted if the failure happened in the middle of the job. By default, this is true. This should be unset if the job has external state (e.g. it modifies a database entry)

  • _type:

    Select the type of compute resources to use. PiCloud supports four types, specified as strings:

    ‘c1’

    1 compute unit, 300 MB ram, low I/O (default)

    ‘c2’

    2.5 compute units, 800 MB ram, medium I/O

    ‘m1’

    3.25 compute units, 8 GB ram, high I/O

    ‘s1’

    variable compute units (2 cu max), 300 MB ram, low I/O, 1 IP per core

    See http://www.picloud.com/pricing/ for pricing information

cloud.iresult(self, jids, timeout=None, num_in_parallel=10, ignore_errors=False)

Similar to result, but returns an iterator that iterates, in order, through the internal results of jids as the respective job finishes, allowing quicker access to results and reducing memory requirements of result reading.

If a job being iterated over errors, an exception is raised. However, the iterator does not exhaust; if the exception is caught, one can continue iterating over the remaining jobs.

If timeout is set to a number, a call to the iterator’s next function will raise a CloudTimeoutError after timeout seconds if no result becomes available.

num_in_parallel controls how many results are read-ahead from the cloud Set this to 0 to use the allowed maximum.

If ignore_errors is True, a job that errored will return the CloudException describing its error, rather than raising an Exception

cloud.info(self, jids, info_requested=None)

Request information about jobs specified by integer or iterable jids

As this function is designed for console debugging, the return value is a dictionary whose keys are the jids requested. The values are themselves dictionaries, whose keys in turn consist of valid values within the iterable info_requested. Each key maps to key-specific information about the job, e.g. stdout maps to standard output of job.

Possible info_requested items are one of more of:

‘status’, ‘stdout’, ‘stderr’, ‘exception’, ‘runtime’- which return job status, standard output,, standard error, exception raised (if applicable) and real runtime of job respectively. If info_requested is None/empty, all info will be requested

The job must have completed for ‘exception’ to be meaningful. All other items can be accessed

while the job is running.

e.g. cloud.info(42,’stdout’) to get standard output of job 42
cloud.kill(self, jids=None)

Kill any incomplete jobs specified by the integer or iterable jids. If no arguments are specified (or jids is None), kill every job that has been submitted.

cloud.delete(self, jids)

Remove all data (result, stdout, etc.) related to jobs specified by the integer or iterable jids Jobs must have finished already to be deleted.

Note

In MP/Simulator mode, this does not delete any datalogs, only in-memory info

cloud.connection_info(self)

Returns a dictionary of information describing the current connection.

cloud.is_simulated(self)

Returns true iff cloud processor is being run on the local machine.

cloud.running_on_cloud(cls)

Returns true iff current thread of execution is running on PiCloud servers.

exception cloud.CloudException

Represents an exception that has occurred by a job on the cloud. CloudException will display the job id for failed cloud requests.

exception cloud.CloudTimeoutError

CloudException specifically for join timeouts

cloud.getconfigpath()

Return the directory where PiCloud configuration and logging files are stored. Configuration/logs are stored on a per-user basis

cloud.close(self)

Terminate cloud connection (or simulation) and all threads utilized by cloud. Returns True if closed cloud; False if cloud was already closed

Note

cloud will automatically re-open if you invoke any cloud.* again

Warning

This function should never be called by a job running on PiCloud

cloud.mp — Multiprocessing Interface

Cloud Multiprocessing Interface

cloud.mp has effectively the same interface as cloud.

All jobs will be run locally, on multiple processors, via the multiprocessing library

Sample usage:

import cloud.mp
jid = cloud.mp.call(lambda: 3*3)
>> Returns a job identifier
cloud.mp.result(jid)
>> Returns 9
cloud.mp.call(self, func, *args, **kwargs)

Invoke func (a callable) in the cloud. When invoked, func will be invoked on PiCloud’s cluster with the passed args that follow it. The invoked function is known as a ‘job’. The return value of the invoked function is known as the ‘internal result’.

Call will return an integer Job IDentifier (jid) which can be passed into the status and result methods to obtain the status of the job and the internal result respectively.

Example:

def mult3(x):
    return 3*x
cloud.call(mult3, 2) 

This will cause mult3 to be invoked on PiCloud’s cluster with x=2

See online documentation for additional information.

Reserved special kwargs (see docs for details):

  • _callback:

    A list of functions that should be run on the callee’s computer once this job finishes successfully.

  • _callback_on_error:

    A list of functions that should be run on the callee’s computer if this job errors.

  • _depends_on:

    An iterable of jids that represents all jobs that must complete successfully before the job created by this call function may be run.

  • _env:

    A string specifying a custom environment you wish to run your job within. See environments overview at http://blog.picloud.com/2011/09/26/introducing-environments-run-anything-on-picloud/

  • _fast_serialization:

    This keyword can be used to speed up serialization, at the cost of some functionality. This affects the serialization of both the arguments and return values func will always be serialized by the enhanced serializer, with debugging features. Possible values keyword are:

    1. default – use cloud module’s enhanced serialization and debugging info
    2. no debug – Disable all debugging features for arguments
    3. use cPickle – Use Python’s fast serializer, possibly causing PicklingErrors
  • _kill_process:

    Terminate the Python interpreter func runs in after func completes, preventing the interpreter from being used by subsequent jobs. See Technical Overview for more info.

  • _label:

    A user-defined string label that is attached to the job. Labels can be used to filter when viewing jobs interactively (i.e. on the PiCloud website).

  • _priority:

    A positive integer denoting the job’s priority. PiCloud tries to run jobs with lower priority numbers before jobs with higher priority numbers.

  • _profile:

    Set this to True to enable profiling of your code. Profiling information is valuable for debugging, but may slow down your job.

  • _restartable:

    In the very rare event of hardware failure, this flag indicates that the job can be restarted if the failure happened in the middle of the job. By default, this is true. This should be unset if the job has external state (e.g. it modifies a database entry)

  • _type:

    Select the type of compute resources to use. PiCloud supports four types, specified as strings:

    ‘c1’

    1 compute unit, 300 MB ram, low I/O (default)

    ‘c2’

    2.5 compute units, 800 MB ram, medium I/O

    ‘m1’

    3.25 compute units, 8 GB ram, high I/O

    ‘s1’

    Up to 2 compute units (variable), 300 MB ram, low I/O, 1 IP per core

    See http://www.picloud.com/pricing/ for pricing information

cloud.mp.join(self, jids, timeout=None, ignore_errors=False)

Block current thread of execution until the job specified by the integer jids completes. Completion is defined as the job finishing or erroring (including stalling). If the job errored, a CloudException detailing the exception that triggered the error is thrown. If multiple errors occur, it is undefined which job’s exception will be raised. If the job does not exist, a CloudException is thrown.

This method also accepts an iterable describing jids and blocks until all corresponding jobs finish. If an error is seen, join may terminate before all jobs finish.

If timeout is set to a number, join will raise a CloudTimeoutError if the job is still running after timeout seconds

If ignore_errors is True, no CloudException will be thrown if a job errored. Join will block until every job is complete, regardless of error status.

cloud.mp.status(self, jids)

Returns the status of the job specified by the integer jids. If the job does not exist, a CloudException is thrown. This method also accepts an iterable describing jids, in which case a respective list of statuses is returned.

cloud.mp.result(self, jids, timeout=None, ignore_errors=False)

Blocks until the job specified by the integer jids has completed and then returns the internal result of the job. If the job errored, a CloudException detailing the exception that triggered the error is thrown. If the job does not exist, a CloudException is thrown.

This function also accepts an iterable describing jids, in which case a respective list of internal results is returned

If timeout is set to a number, result will raise a CloudTimeoutError if the job is still running after timeout seconds

If ignore_errors is True, a job that errored will not raise an exception. Instead, its return value is the CloudException describing the error.

cloud.mp.map(self, func, *args, **kwargs)

Map func (a callable) over argument sequence(s). cloud.map is meant to mimic a regular map call such as:

map(lambda x,y: x*y, xlist, ylist)

args can be any number of iterables. If the iterables are of different size, ‘None’ is substituted for a missing entries.

Map will return an iterable describing integer Job IDentifiers (jids). Each jid corresponds to func being invoked on one item of the sequence(s) (a ‘job’). In practice, the jids can (and should) be treated as a single jid; the returned iterable may be passed directly into status, result, etc.

Using cloud.result on the returned jids will return a list of internal results (each being the result of applying the function to an item of the argument sequence(s)).

Example:

def mult(x,y):
    return x*y
jids = cloud.map(mult, [1,3,5], [2,4,6]) 

This will cause mult3 to be invoked on PiCloud’s cluster with x=2

Results:

cloud.result(jids)
>> [2,12,30]

The result is [1*2,3*4,5*6]

See online documentation for additional information

Reserved special kwargs (see docs for details):

  • _depends_on:

    An iterable of jids that represents all jobs that must complete successfully before any jobs created by this map function may be run.

  • _env:

    A string specifying a custom environment you wish to run your jobs within. See environments overview at http://blog.picloud.com/2011/09/26/introducing-environments-run-anything-on-picloud/

  • _fast_serialization:

    This keyword can be used to speed up serialization, at the cost of some functionality. This affects the serialization of both the map arguments and return values The map function will always be serialized by the enhanced serializer, with debugging features. Possible values keyword are:

    1. default – use cloud module’s enhanced serialization and debugging info
    2. no debug – Disable all debugging features for arguments
    3. use cPickle – Use python’s fast serializer, possibly causing PicklingErrors
  • _kill_process:

    Terminate the Python interpreter func runs in after func completes, preventing the interpreter from being used by subsequent jobs. See Technical Overview for more info.

  • _label:

    A user-defined string label that is attached to the created jobs. Labels can be used to filter when viewing jobs interactively (i.e. on the PiCloud website).

  • _priority:

    A positive integer denoting the job’s priority. PiCloud tries to run jobs with lower priority numbers before jobs with higher priority numbers.

  • _profile:

    Set this to True to enable profiling of your code. Profiling information is valuable for debugging, but may slow down your job.

  • _restartable:

    In the very rare event of hardware failure, this flag indicates that the job can be restarted if the failure happened in the middle of the job. By default, this is true. This should be unset if the job has external state (e.g. it modifies a database entry)

  • _type:

    Select the type of compute resources to use. PiCloud supports four types, specified as strings:

    ‘c1’

    1 compute unit, 300 MB ram, low I/O (default)

    ‘c2’

    2.5 compute units, 800 MB ram, medium I/O

    ‘m1’

    3.25 compute units, 8 GB ram, high I/O

    ‘s1’

    variable compute units (2 cu max), 300 MB ram, low I/O, 1 IP per core

    See http://www.picloud.com/pricing/ for pricing information

cloud.mp.iresult(self, jids, timeout=None, num_in_parallel=10, ignore_errors=False)

Similar to result, but returns an iterator that iterates, in order, through the internal results of jids as the respective job finishes, allowing quicker access to results and reducing memory requirements of result reading.

If a job being iterated over errors, an exception is raised. However, the iterator does not exhaust; if the exception is caught, one can continue iterating over the remaining jobs.

If timeout is set to a number, a call to the iterator’s next function will raise a CloudTimeoutError after timeout seconds if no result becomes available.

num_in_parallel controls how many results are read-ahead from the cloud Set this to 0 to use the allowed maximum.

If ignore_errors is True, a job that errored will return the CloudException describing its error, rather than raising an Exception

cloud.mp.info(self, jids, info_requested=None)

Request information about jobs specified by integer or iterable jids

As this function is designed for console debugging, the return value is a dictionary whose keys are the jids requested. The values are themselves dictionaries, whose keys in turn consist of valid values within the iterable info_requested. Each key maps to key-specific information about the job, e.g. stdout maps to standard output of job.

Possible info_requested items are one of more of:

‘status’, ‘stdout’, ‘stderr’, ‘exception’, ‘runtime’- which return job status, standard output,, standard error, exception raised (if applicable) and real runtime of job respectively. If info_requested is None/empty, all info will be requested

The job must have completed for ‘exception’ to be meaningful. All other items can be accessed

while the job is running.

e.g. cloud.info(42,’stdout’) to get standard output of job 42
cloud.mp.kill(self, jids=None)

Kill any incomplete jobs specified by the integer or iterable jids. If no arguments are specified (or jids is None), kill every job that has been submitted.

cloud.mp.delete(self, jids)

Remove all data (result, stdout, etc.) related to jobs specified by the integer or iterable jids Jobs must have finished already to be deleted.

Note

In MP/Simulator mode, this does not delete any datalogs, only in-memory info

cloud.mp.connection_info(self)

Returns a dictionary of information describing the current connection.

cloud.mp.close(self)

Terminate cloud connection (or simulation) and all threads utilized by cloud. Returns True if closed cloud; False if cloud was already closed

Note

cloud will automatically re-open if you invoke any cloud.* again

Warning

This function should never be called by a job running on PiCloud

cloud.files — PiCloud File Storage Interface

For managing files on PiCloud’s S3 store.

Api keys must be configured before using any functions in this module. (via cloudconf.py, cloud.setkey, or being on PiCloud server)

Note

This module cannot be used to access files stored on a job’s mounted file system

cloud.files.put(source, name=None)

Transfers the file specified by file_path to PiCloud. The file can be retrieved later using the get function.

If name is specified, the file will be stored as name on PiCloud. Otherwise it will be stored as the basename of file_path.

Example:

cloud.files.put('data/names.txt') 

This will transfer the file from the local path ‘data/names.txt’ to PiCloud and store it as ‘names.txt’. It can be later retrieved via cloud.files.get(‘names.txt’)

cloud.files.sync_to_cloud(source, name=None)

Upload file if it has changed.

cloud.files.put(source,name) only if contents of local file (specified by source) differ from those on PiCloud (specified by name or basename(source)) (or if file does not exist on PiCloud)

cloud.files.putf(f, name)

Similar to put. putf, however, accepts a file object (file, StringIO, etc.) f instead of a file_path.

Note

f is not rewound. f.read() from current position will be placed on PiCloud

Warning

If the file object does not correspond to an actual file on disk, it will be read entirely into memory before being transferred to PiCloud.

cloud.files.list()

List all files stored on PiCloud.

cloud.files.exists(name)

Check if a file named name is stored on PiCloud.

cloud.files.delete(name)

Deletes the file named name from PiCloud.

cloud.files.get(file_name, save_path=None, start_byte=0, end_byte=None)

Retrieves the file named by file_name from PiCloud and stores it to save_path.

Example:

cloud.files.get('names.txt','data/names.txt') 

This will retrieve the ‘names.txt’ file on PiCloud and save it locally to ‘data/names.txt’.

If save_path is None, save_path will be file_name

An optional byte_range can be specified using start_byte, end_byte, where only the data between start_byte and end_byte is returned. If end_byte exceeds the size of the file, the contents from start_byte to end of file returned.

An end_byte of None or exceeding file size is interpreted as end of file

cloud.files.getf(file_name, start_byte=0, end_byte=None)

Retrieves the file named by file_name from PiCloud. Return value is a CloudFile (file-like object) that can be read() to retrieve the file’s contents

A range can be specified through start_byte and end_byte, where only the data between those two offsets will be accessable in the CloudFile. If start_byte is set, the returned CloudFile.tell() will be start_byte

An end_byte of None or exceeding file size is interpretted as end of file

cloud.files.sync_from_cloud(name, destination=None)

Download file if it has changed.

cloud.files.get(name,destination) only if contents of local file (specified by destination or basename(file_name)) differ from those on PiCloud (specified by *name) (or destination does not exist locally)

cloud.files.get_md5(name, log_missing_file_error=True)

Return the md5 checksum of the file named name stored on PiCloud

class cloud.files.CloudFile(http_response, file_size, start_byte=0, end_byte=None)

A CloudFile represents a file stored on PiCloud as a readonly file-like stream. Seeking is not available.

close()

Close the file, blocking further I/O operations

md5

The md5 checksum of file contents

read(size=-1)

read([size]) -> read at most size bytes, returned as a string.

If the size argument is negative or omitted, read until EOF is reached.

readline(size=-1)

readline([size]) -> next line from the file, as a string.

Retain newline. A non-negative size argument limits the maximum number of bytes to return (an incomplete line may be returned then). Return an empty string at EOF.

readlines(sizehint=0)

readlines([size]) -> list of strings, each a line from the file.

Call readline() repeatedly and return a list of the lines so read. The optional size argument, if given, is an approximate bound on the total number of bytes in the lines returned.

tell()

Returns current file position as an integer

cloud.cron — PiCloud Cron Interface

For managing crons on PiCloud. Crons allow you to “register” a function to be invoked periodically on PiCloud according to a schedule you specify.

Api keys must be configured before using any functions in this module. (via cloudconf.py, cloud.setkey, or being on PiCloud server)

cloud.cron.register(func, label, schedule, **kwargs)

Register func (a callable) to be run periodically on PiCloud according to schedule

The cron can be managed in the future by the specified label

Schedule is a BSD-style cron timestamp, specified as either a string of 5 non- whitespace characters delimited by spaces or as a sequence (tuple, list, etc.) of 5 elements, each with non-whitespace characters. The ordering of the characters represent scheduling information for minutes, hours, days, months, and days of week. See full format specification at http://unixhelp.ed.ac.uk/CGI/man-cgi?crontab+5 Year is not supported. The N/S increment format is though.

PiCloud schedules your crons under the GMT (UTC+0) timezone.

Certain special kwargs associated with cloud.call can be attached to the periodic jobs:

  • _env:

    A string specifying a custom environment jobs should use See environments overview at http://blog.picloud.com/2011/09/26/introducing-environments-run-anything-on-picloud/

  • _fast_serialization:

    This keyword can be used to speed up serialization, at the cost of some functionality. This affects the serialization of the spawned jobs’ return value. The stored function will always be serialized by the enhanced serializer, with debugging features. Possible values keyword are:

    1. default – use cloud module’s enhanced serialization and debugging info
    2. no debug – Disable all debugging features for result
    3. use cPickle – Use python’s fast serializer, possibly causing PicklingErrors
  • _profile:

    Set this to True to enable profiling of your code. Profiling information is valuable for debugging, but may slow down your jobs.

  • _restartable:

    In the very rare event of hardware failure, this flag indicates that a spawned job can be restarted if the failure happened in the middle of it. By default, this is true. This should be unset if the function has external state (e.g. it modifies a database entry)

  • _type:

    Select the type of compute resources to use. PiCloud supports four types, specified as strings:

    ‘c1’

    1 compute unit, 300 MB ram, low I/O (default)

    ‘c2’

    2.5 compute units, 800 MB ram, medium I/O

    ‘m1’

    3.25 compute units, 8 GB ram, high I/O

    ‘s1’

    Up to 2 compute units (variable), 300 MB ram, low I/O, 1 IP per core

    See http://www.picloud.com/pricing/ for pricing information

Note

The callable must take no arguments. To schedule functions that take arguments, wrap it with a closure. e.g. func(x,y) –> lambda: func(x,y)

cloud.cron.deregister(label)

Deregister (delete) the cron specified by label

cloud.cron.manual_run(label)

Manually run the cron specified by label Returns the ‘jid’ of the job created by this run command

cloud.cron.list()

List labels of active crons

cloud.cron.info(label)

Return a dictionary of relevant info about cron specified by label

Info includes:

  • label: The same as was passed in
  • schedule: The schedule of this cron
  • last_run: Time this cron was last run
  • last_jid: Last run jid of this cron
  • created: When this cron was created
  • creator_host: Hostname of the machine that registered this cron
  • funcname: Name of function associated with cron

cloud.rest — PiCloud REST Registration Interface

Interface to publish functions to PiCloud, allowing them to be invoked via the REST API

Api keys must be configured before using any functions in this module. (via cloudconf.py, cloud.setkey, or being on PiCloud server)

cloud.rest.publish(func, label, out_encoding='json', **kwargs)

Publish func (a callable) to PiCloud so it can be invoked through the PiCloud REST API

The published function will be managed in the future by a unique (URL encoded) label.

out_encoding specifies the format that the return value should be in when retrieving the result via the REST API. Valid values are “json” for a JSON-encoded object and RAW, where the return value must be an str (but can contain any characters).

The return value is the URL which can be HTTP POSTed to to invoke func. See http://docs.picloud.com/rest.html for information about PiCloud’s REST API

Certain special kwargs associated with cloud.call can be attached to the periodic jobs:

  • _fast_serialization:

    This keyword can be used to speed up serialization, at the cost of some functionality. This affects the serialization of the spawned jobs’ return value. The stored function will always be serialized by the enhanced serializer, with debugging features. Possible values keyword are:

    1. default – use cloud module’s enhanced serialization and debugging info
    2. no debug – Disable all debugging features for result
    3. use cPickle – Use python’s fast serializer, possibly causing PicklingErrors
  • _env:

    A string specifying a custom environment you wish to run your generated jobs within. See environments overview at http://blog.picloud.com/2011/09/26/introducing-environments-run-anything-on-picloud/

  • _priority:

    A positive integer denoting the job’s priority. PiCloud tries to run jobs with lower priority numbers before jobs with higher priority numbers.

  • _profile:

    Set this to True to enable profiling of your code. Profiling information is valuable for debugging, but may slow down your jobs.

  • _restartable:

    In the very rare event of hardware failure, this flag indicates that a spawned job can be restarted if the failure happened in the middle of it. By default, this is true. This should be unset if the function has external state (e.g. it modifies a database entry)

  • _type:

    Select the type of compute resources to use. PiCloud supports four types, specified as strings:

    ‘c1’

    1 compute unit, 300 MB ram, low I/O (default)

    ‘c2’

    2.5 compute units, 800 MB ram, medium I/O

    ‘m1’

    3.25 compute units, 8 GB ram, high I/O

    ‘s1’

    Up to 2 compute units (variable), 300 MB ram, low I/O, 1 IP per core

    See http://www.picloud.com/pricing/ for pricing information

cloud.rest.remove(label)

Remove a published function from PiCloud

cloud.rest.list()

List labels of published functions

cloud.rest.info(label)

Retrieve information about a published function specified by label

cloud.config — Configuration Interface

class cloud.util.configmanager.ConfigSettings(confmanager, do_reload=False)

This object provides the ability to programmatically edit the cloud configuration (found in cloudconf.py). commit() must be called to update the cloud module with new settings - and restart all active clouds

api_key

your application’s api key provided by PiCloud

cloud_result_cache_size

Amount (in kb) of job results to hold in local memory; set to 0 to disable caching. This option only applies when connected to PiCloud.

cloud_status_cache_size

Number of job statuses to hold in local memory; set to 0 to disable caching. This option only applies when connected to PiCloud.

commit()

Update cloud with new settings.

Warning

This will restart any active cloud instances, wiping mp/simulated jobs and setkey information

force_redirect_job_output

If set and cloud is in simulation mode, forces redirect_job_output (see multiprocessing section) to true

force_serialize_logging

If this is is set and cloud is running in simulation mode, serialize_logging (see logging section) will be turned on

http_close

Should every HTTP connection be closed after receiving a response?

log_filename

Filename where cloud log messages should be written. This path is relative to ~/.picloud/

log_level

logging level for cloud messages. This affects both messages saved to the cloud log file and sent through the python logging system. See http://docs.python.org/library/logging.html for more information

max_transmit_data

Maximum amount of data (in bytes) that can be sent to the PiCloud server per function or function argument. Value may be raised up to 16 MB

mp_cache_size_limit

Maxmimum number of jobs that cloudmp should store. Set to 0 for no limit. This option only applies to cloud.mp and the cloud simulator. PiCloud recommends that any limit stays above 1024.

num_procs

Number of subprocesses that cloud multiprocessing should utilize. Beware of using too low of a number of subprocesses, as deadlock can occur. If set to 0, this will be set to the number of cores available on this system.

print_log_level

logging level for printing cloud log messages to console. Must be equal or higher than log_level

proxy_server

Specify an http/https proxy server that should be used e.g. proxy.example.com:3128 for anonymous or username:password@proxy.example.com:3128 for authenticated

purge_days

Number of days to wait until purging old serialization log directories

redirect_job_output

If set to true, job stdout and stderr will be redirected to files inside the serialize_logging_path. This option simulates PiCloud behavior and must be set to true for cloud.info to be able to request stdout or stderr information about jobs. If set to false, job stdout and stderr will be the same as the parent process.

save_log

Should log_filename (default of cloud.log) be written with cloud log messages? Note that cloud will always emit logging messages; this option controls if cloud should have its own log.

serialize_logging

Should all object serialization meta-data be logged to the serialize_logging_path?

serialize_logging_path

Path to save object serialization meta-data. This path is relative to ~/.picloud/

server_list_url

url to list of PiCloud servers

use_gzip

Request gziped HTTP responses

use_simulator

The Cloud Simulator simulates cloud.* functions locally, allowing for quicker debugging. If this is enabled, all cloud.* functions will be run locally, rather than on PiCloud. See web documentation.

cloud.account — Account Management Interface

PiCloud account management. This module allows the user to manage account information programmatically. Currently allows managing real time core requests.

Note that your api_key must be set before using any functions.

cloud.account.list_keys(username, password, active_only=False)

Returns a list of all api keys. If active_only is True, only active keys are returned. username and password should be your PiCloud login information.

cloud.account.get_key(username, password, api_key)

Returns information including api_secretkey, active status, and note for the specified api_key. username and password should be your PiCloud login information.

cloud.account.activate_key(username, password, api_key)

Activates the specified api_key. username and password should be your PiCloud login information.

cloud.account.deactivate_key(username, password, api_key)

Deactivates the specified api_key. username and password should be your PiCloud login information.

cloud.account.create_key(username, password)

Deactivates the specified api_key. username and password should be your PiCloud login information.

cloud.pool_interface — Multiprocessing Pool Wrapper

A module provided that emulates python multiprocessing’s Process Pool interface. This can be used to trivially replace multiprocessing Pool code.

Note that cloud.setkey must still be used before using this interface.

Not supported (mostly due to not making sense in the PiCloud context).

  • Pool initializer
  • Passing keyword arguments to function being applied
  • callback for map/map_async
  • (i)map chunksize – argument only controls cloud.iresult chunksize
  • iresult_undordered – this is just an ordered iresult
  • multiprocessing.TimeoutError – cloud.CloudTimeoutError is used instead
  • join, close and terminate operations

Warning

It is highly recommended that you do not use this module and instead use the cloud module directly. Much functionality that the cloud interface offers is missing here.

cloud.pool_interface.apply(func, args=())

Equivalent to Multiprocessing apply. keyword arguments are not supported

cloud.pool_interface.apply_async(func, args=(), callback=None)

Equivalent to Multiprocessing apply_async keyword arguments are not supported

cloud.pool_interface.map(func, iterable, chunksize=None)

Equivalent to Multiprocessing map chunksize is not used here

cloud.pool_interface.map_async(func, iterable, chunksize=None)

Equivalent to Multiprocessing map_async chunksize is not used here

cloud.pool_interface.imap(func, iterable, chunksize=None)

Equivalent to Multiprocessing imap chunksize is used only to control the cloud.iresult stage

cloud.pool_interface.imap_unordered(func, iterable, chunksize=None)

Same as imap

class cloud.pool_interface.AsyncResult(jid)

Result object that emulates multiprocessing.pool.AsyncResult

get(timeout=None)

Return result when it arrives. If timeout is not None and none arrives, raise multiprocessing.TimeoutError in timeout seconds

ready()

Returns true if the job finished (done or errored)

successful()

Returns true if job finished successfully. Asserts that job has finished

wait(timeout=None)

Wait until result is available or timeout seconds pass