Interface for cloud - the clientside module for PiCloud
Sample usage:
import cloud
jid = cloud.call(lambda: 3*3)
>> Returns a job identifier
cloud.result(jid)
>> Returns 9
This will run the function lambda: 3*3 on PiCloud’s cluster, and return the result. Most functions, even user-defined ones, can be passed through cloud.call
For cloud to work, you must first do one the following:
Connect cloud to your PiCloud account, Your api_key and api_secretkey are provided by PiCloud on the API Keys section of the PiCloud website.
server_url specifies the PiCloud server to connect to. Leave this blank to have cloud auto-resolve servers.
restart forces the cloud to reconnect
This command will disable the simulator if it is running.
Invoke func (a callable) in the cloud. When invoked, func will be invoked on PiCloud’s cluster with the passed args that follow it. The invoked function is known as a ‘job’. The return value of the invoked function is known as the ‘internal result’.
Call will return an integer Job IDentifier (jid) which can be passed into the status and result methods to obtain the status of the job and the internal result respectively.
Example:
def mult3(x):
return 3*x
cloud.call(mult3, 2)
This will cause mult3 to be invoked on PiCloud’s cluster with x=2
See online documentation for additional information.
Reserved special kwargs (see docs for details):
A list of functions that should be run on the callee’s computer once this job finishes successfully.
A list of functions that should be run on the callee’s computer if this job errors.
An iterable of jids that represents all jobs that must complete successfully before the job created by this call function may be run.
This keyword can be used to speed up serialization, at the cost of some functionality. This affects the serialization of both the arguments and return values func will always be serialized by the enhanced serializer, with debugging features. Possible values keyword are:
Set this to true to run your job on a faster processor. Additional charges apply.
Terminate the Python interpreter func runs in after func completes, preventing the interpreter from being used by subsequent jobs. See Technical Overview for more info.
A user-defined string label that is attached to the job. Labels can be used to filter when viewing jobs interactively (i.e. on the PiCloud website).
Set this to True to enable profiling of your code. Profiling information is valuable for debugging, but may slow down your job.
In the very rare event of hardware failure, this flag indicates that the job can be restarted if the failure happened in the middle of the job. By default, this is true. This should be unset if the job has external state (e.g. it modifies a database entry)
Block current thread of execution until the job specified by the integer jids completes. Completion is defined as the job finishing or erroring (including stalling). If the job errored, a CloudException detailing the exception that triggered the error is thrown. If the job does not exist, a CloudException is thrown.
This method also accepts an iterable describing jids and blocks until all jids finish or error.
If timeout is set to a number, join will raise a CloudTimeoutError if the job is still running after timeout seconds
Blocks until the job specified by the integer jids has completed and then returns the internal result of the job. If the job errored, a CloudException detailing the exception that triggered the error is thrown. If the job does not exist, a CloudException is thrown.
This function also accepts an iterable describing jids, in which case a respective list of internal results is returned
If timeout is set to a number, result will raise a CloudTimeoutError if the job is still running after timeout seconds
Map func (a callable) over argument sequence(s). cloud.map is meant to mimic a regular map call such as:
map(lambda x,y: x*y, xlist, ylist)
args can be any number of iterables. If the iterables are of different size, ‘None’ is substituted for a missing entries.
Map will return an iterable describing integer Job IDentificatiers (jids). Each jid corresponds to func being invoked on one item of the sequence(s) (a ‘job’). In practice, the jids can (and should) be treated as a single jid; the returned iterable may be passed directly into status, result, etc.
Using cloud.result on the returned jids will return a list of internal results (each being the result of applying the function to an item of the argument sequence(s)).
Example:
def mult(x,y):
return x*y
jids = cloud.map(mult, [1,3,5], [2,4,6])
This will cause mult3 to be invoked on PiCloud’s cluster with x=2
Results:
cloud.result(jids)
>> [2,12,30]
The result is [1*2,3*4,5*6]
See online documentation for additional information
Reserved special kwargs (see docs for details):
An iterable of jids that represents all jobs that must complete successfully before any jobs created by this map function may be run.
This keyword can be used to speed up serialization, at the cost of some functionality. This affects the serialization of both the map arguments and return values The map function will always be serialized by the enhanced serializer, with debugging features. Possible values keyword are:
Set this to true to run your job on a faster processor. Additional charges apply.
Terminate the Python interpreter func runs in after func completes, preventing the interpreter from being used by subsequent jobs. See Technical Overview for more info.
A user-defined string label that is attached to the created jobs. Labels can be used to filter when viewing jobs interactively (i.e. on the PiCloud website).
Set this to True to enable profiling of your code. Profiling information is valuable for debugging, but may slow down your job.
In the very rare event of hardware failure, this flag indicates that the job can be restarted if the failure happened in the middle of the job. By default, this is true. This should be unset if the job has external state (e.g. it modifies a database entry)
Similar to result, but returns an iterator that iterates, in order, through the internal results of jids as the respective job finishes, allowing quicker access to results and reducing memory requirements of result reading.
If a job being iterated over errors, an exception is raised. However, the iterator does not exhaust; if the exception is caught, one can continue iterating over the remaining jobs.
If timeout is set to a number, a call to the iterator’s next function will raise a CloudTimeoutError after timeout seconds if no result becomes available.
num_in_parallel controls how many results are read-ahead from the cloud Set this to 0 to use the allowed maximum.
Request information about jobs specified by integer or iterable jids
As this function is designed for console debugging, the return value is a dictionary whose keys are the jids requested. The values are themselves dictionaries, whose keys in turn consist of valid values within the iterable info_requested. Each key maps to key-specific information about the job, e.g. stdout maps to standard output of job.
‘stdout’, ‘stderr’, ‘exception’, ‘runtime’- which return standard output, standard error, exception raised (if applicable) and real runtime of job respectively.
e.g. cloud.info(42,’stdout’) to get standard output of job 42
Remove all data (result, stdout, etc.) related to jobs specified by the integer or iterable jids Jobs must have finished already to be deleted.
Note
In MP/Simulator mode, this does not delete any datalogs, only in-memory info
Cloud Multiprocessing Interface
cloud.mp has effectively the same interface as cloud.
All jobs will be run locally, on multiple processors, via the multiprocessing library
Sample usage:
import cloud.mp
jid = cloud.mp.call(lambda: 3*3)
>> Returns a job identifier
cloud.mp.result(jid)
>> Returns 9
Invoke func (a callable) in the cloud. When invoked, func will be invoked on PiCloud’s cluster with the passed args that follow it. The invoked function is known as a ‘job’. The return value of the invoked function is known as the ‘internal result’.
Call will return an integer Job IDentifier (jid) which can be passed into the status and result methods to obtain the status of the job and the internal result respectively.
Example:
def mult3(x):
return 3*x
cloud.call(mult3, 2)
This will cause mult3 to be invoked on PiCloud’s cluster with x=2
See online documentation for additional information.
Reserved special kwargs (see docs for details):
A list of functions that should be run on the callee’s computer once this job finishes successfully.
A list of functions that should be run on the callee’s computer if this job errors.
An iterable of jids that represents all jobs that must complete successfully before the job created by this call function may be run.
This keyword can be used to speed up serialization, at the cost of some functionality. This affects the serialization of both the arguments and return values func will always be serialized by the enhanced serializer, with debugging features. Possible values keyword are:
Set this to true to run your job on a faster processor. Additional charges apply.
Terminate the Python interpreter func runs in after func completes, preventing the interpreter from being used by subsequent jobs. See Technical Overview for more info.
A user-defined string label that is attached to the job. Labels can be used to filter when viewing jobs interactively (i.e. on the PiCloud website).
Set this to True to enable profiling of your code. Profiling information is valuable for debugging, but may slow down your job.
In the very rare event of hardware failure, this flag indicates that the job can be restarted if the failure happened in the middle of the job. By default, this is true. This should be unset if the job has external state (e.g. it modifies a database entry)
Block current thread of execution until the job specified by the integer jids completes. Completion is defined as the job finishing or erroring (including stalling). If the job errored, a CloudException detailing the exception that triggered the error is thrown. If the job does not exist, a CloudException is thrown.
This method also accepts an iterable describing jids and blocks until all jids finish or error.
If timeout is set to a number, join will raise a CloudTimeoutError if the job is still running after timeout seconds
Blocks until the job specified by the integer jids has completed and then returns the internal result of the job. If the job errored, a CloudException detailing the exception that triggered the error is thrown. If the job does not exist, a CloudException is thrown.
This function also accepts an iterable describing jids, in which case a respective list of internal results is returned
If timeout is set to a number, result will raise a CloudTimeoutError if the job is still running after timeout seconds
Map func (a callable) over argument sequence(s). cloud.map is meant to mimic a regular map call such as:
map(lambda x,y: x*y, xlist, ylist)
args can be any number of iterables. If the iterables are of different size, ‘None’ is substituted for a missing entries.
Map will return an iterable describing integer Job IDentificatiers (jids). Each jid corresponds to func being invoked on one item of the sequence(s) (a ‘job’). In practice, the jids can (and should) be treated as a single jid; the returned iterable may be passed directly into status, result, etc.
Using cloud.result on the returned jids will return a list of internal results (each being the result of applying the function to an item of the argument sequence(s)).
Example:
def mult(x,y):
return x*y
jids = cloud.map(mult, [1,3,5], [2,4,6])
This will cause mult3 to be invoked on PiCloud’s cluster with x=2
Results:
cloud.result(jids)
>> [2,12,30]
The result is [1*2,3*4,5*6]
See online documentation for additional information
Reserved special kwargs (see docs for details):
An iterable of jids that represents all jobs that must complete successfully before any jobs created by this map function may be run.
This keyword can be used to speed up serialization, at the cost of some functionality. This affects the serialization of both the map arguments and return values The map function will always be serialized by the enhanced serializer, with debugging features. Possible values keyword are:
Set this to true to run your job on a faster processor. Additional charges apply.
Terminate the Python interpreter func runs in after func completes, preventing the interpreter from being used by subsequent jobs. See Technical Overview for more info.
A user-defined string label that is attached to the created jobs. Labels can be used to filter when viewing jobs interactively (i.e. on the PiCloud website).
Set this to True to enable profiling of your code. Profiling information is valuable for debugging, but may slow down your job.
In the very rare event of hardware failure, this flag indicates that the job can be restarted if the failure happened in the middle of the job. By default, this is true. This should be unset if the job has external state (e.g. it modifies a database entry)
Similar to result, but returns an iterator that iterates, in order, through the internal results of jids as the respective job finishes, allowing quicker access to results and reducing memory requirements of result reading.
If a job being iterated over errors, an exception is raised. However, the iterator does not exhaust; if the exception is caught, one can continue iterating over the remaining jobs.
If timeout is set to a number, a call to the iterator’s next function will raise a CloudTimeoutError after timeout seconds if no result becomes available.
num_in_parallel controls how many results are read-ahead from the cloud Set this to 0 to use the allowed maximum.
Request information about jobs specified by integer or iterable jids
As this function is designed for console debugging, the return value is a dictionary whose keys are the jids requested. The values are themselves dictionaries, whose keys in turn consist of valid values within the iterable info_requested. Each key maps to key-specific information about the job, e.g. stdout maps to standard output of job.
‘stdout’, ‘stderr’, ‘exception’, ‘runtime’- which return standard output, standard error, exception raised (if applicable) and real runtime of job respectively.
e.g. cloud.info(42,’stdout’) to get standard output of job 42
Remove all data (result, stdout, etc.) related to jobs specified by the integer or iterable jids Jobs must have finished already to be deleted.
Note
In MP/Simulator mode, this does not delete any datalogs, only in-memory info
For managing files on PiCloud.
Note that cloud.setkey() must be called before using any functions.
Transfers the file specified by file_path to PiCloud. The file can be retrieved later using the get function.
If name is specified, the file will be stored as name on PiCloud. Otherwise it will be stored as the basename of file_path.
Example:
cloud.files.put('data/names.txt')
This will transfer the file from the local path ‘data/names.txt’ to PiCloud and store it as ‘names.txt’. It can later retrieved via cloud.files.get(‘names.txt’)
Similar to put. putf, however, accepts a file object f instead of a file_path.
Warning
If the file object does not correspond to an actual file on disk, it will be read entirely into memory before being transferred to PiCloud.
Retrieves the file named by file_name from PiCloud and stores it to save_path.
Example:
cloud.files.get('names.txt','data/names.txt')
This will retrieve the ‘names.txt’ file on PiCloud and save it locally to ‘data/names.txt’.
For managing crons on PiCloud. Crons allow you to “register” a function to be invoked periodically on PiCloud according to a schedule you specify.
Note that api_key must be set before using any functions.
Register func (a callable) to be run periodically on PiCloud according to schedule
The cron can be managed in the future by the specified label
Schedule is a BSD-style cron timestamp, specified as either a string of 5 non- whitespace characters delimited by spaces or as a sequence (tuple, list, etc.) of 5 elements, each with non-whitespace characters. The ordering of the characters represent scheduling information for minutes, hours, days, months, and days of week. See full format specification at http://unixhelp.ed.ac.uk/CGI/man-cgi?crontab+5 Year is not supported. The N/S increment format is though.
PiCloud schedules your crons under the GMT (UTC+0) timezone.
Certain special kwargs associated with cloud.call can be attached to the periodic jobs:
This keyword can be used to speed up serialization, at the cost of some functionality. This affects the serialization of the spawned jobs’ return value. The stored function will always be serialized by the enhanced serializer, with debugging features. Possible values keyword are:
Set this to true to run jobs on a faster processor. Additional charges may apply.
Set this to True to enable profiling of your code. Profiling information is valuable for debugging, but may slow down your jobs.
In the very rare event of hardware failure, this flag indicates that a spawned job can be restarted if the failure happened in the middle of it. By default, this is true. This should be unset if the function has external state (e.g. it modifies a database entry)
Note
The callable must take no arguments. To schedule functions that take arguments, wrap it with a closure. e.g. func(x,y) –> lambda: func(x,y)
Return a dictionary of relevant info about cron specified by label
Info includes:
This object provides the ability to programmatically edit the cloud configuration (found in cloudconf.py). commit() must be called to update the cloud module with new settings - and restart all active clouds
Update cloud with new settings.
Warning
This will restart any active cloud instances, wiping mp/simulated jobs and setkey information
Python based account access. This module allows the user to provision API keys programmatically
Note that setuser() must be called before using any functions
Returns a list of dictionaries, each which defines an api_key. The keys within each return api_key are:
Create an api key, which will automatically be activated. Returns information about the newly created key.
Note
There is a limit of 50 active api keys
A module provided that emulates python multiprocessing’s Process Pool interface. This can be used to trivially replace multiprocessing Pool code.
Note that cloud.setkey must still be used before using this interface.
Not supported (mostly due to not making sense in the PiCloud context).
Warning
It is highly recommended that you do not use this module and instead use the cloud module directly. Much functionality that the cloud interface offers is missing here.
Result object that emulates multiprocessing.pool.AsyncResult