REST API

PiCloud’s RESTful API lets you use PiCloud directly over HTTP. It’s useful if you need to utilize PiCloud without our Python library, cloud, or our command-line interface, picloud. For example, Google App Engine does not support the installation of our cloud library (picloud doesn’t work without cloud), so you must use REST. Another use case of the REST API is to create Webhooks that leverage PiCloud.

The interface of PiCloud’s REST API is designed to be similar to the interfaces available in cloud and picloud. However, there is no equivalent to cloud.call() and picloud exec for function invocation. The REST analog for these is a two-steps process:

  1. Publish a function using cloud.rest or picloud rest
  2. Make a POST request to the associated URL to create a job.

Throughout this article, we will be showing example queries made using curl. You can make similar requests, substituting key with your api key and secret_key with your api secret key.

Query Format

All queries to the REST API must be made over HTTPS (unencrypted HTTP is not supported) to the PiCloud API server, located at https://api.picloud.com.

Requests are authenticated with HTTP Basic Authentication. The username is your api key and the password is your api secret key. Unless otherwise noted, responses will be a JSON encoded dictionary containing query-specific information.

curl

This is the recommended way for getting familiar with our REST API.

$ curl -k -u 'key:secret_key' https://api.picloud.com/servers/list/
{"servers": ["https://api.picloud.com/"]}

For a user with key 1337 and secret key of abcd, the request would be:

$ curl -k -u '1337:abcd' https://api.picloud.com/servers/list/
{"servers": ["https://api.picloud.com/"]}

Python example

Here’s an example of how you’d make a REST call in Python.

import base64
import json
import urllib2

base64string = base64.encodestring('%s:%s' % (key, secret_key))[:-1]  #[:-1] removes newline at end
http_headers = {'Authorization' : 'Basic %s' % base64string}          #HTTP basic authentication

request = urllib2.Request('https://api.picloud.com/servers/list/', post_values={}, http_headers)
response = urllib2.urlopen(request)
data = json.load(response)   #{"servers": ["https://api.picloud.com/"]}

Google App Engine

Because App Engine does not let you use standard Python libraries such as urllib2, you need to use their urlfetch library when making REST calls:

import base64
import json
import urlfetch

base64string = base64.encodestring('%s:%s' % (key, secret_ley))[:-1]
http_headers = {'Authorization' : 'Basic %s' % base64string}

response = urlfetch.fetch(url='https://api.picloud.com/servers/list/',
          payload={},  #POST DATA if method was urlfetch.POST
          method=urlfetch.GET,
          headers=http_headers)
data = json.loads(response.content)    #{"servers": ["https://api.picloud.com/"]}

Running Jobs

Publishing functions

A Python callable (function) is published by passing it into cloud.rest.publish(). A URL, dictated by the unique label you specify, will be returned. An example:

>>> def square(x):
...    """Returns square of a number"""
...    print 'Squaring %d' % x
...    return x*x
>>>
>>> cloud.rest.publish(square, 'square_func')
'https://api.picloud.com/r/unique_id/square_func'

Shell commands may be similarly published:

$ picloud rest publish hello-to echo hello, {name}
https://api.picloud.com/r/unique_id/hello-to

Warning

Labels are unique. If a previously published function labeled square_func exists, it will be removed to make way for a new entry.

Retrieving Information

Make an HTTP GET request to the URL returned by cloud.rest.publish() to obtain information about the published function, including its label, encoding type of the return value, URL to access it, signature showing its parameters, and its description (docstring of the function).

To retrieve information about square_func:

$ curl -k -u 'key:secret_key' https://api.picloud.com/r/unique_id/square_func/
{
    "label": "square_func",
    "output_encoding": "json",
    "uri": "https://api.picloud.com/r/unique_id/square_func",
    "signature": "square_func(x)"
    "description": "Returns square of a number"
}

Or the shell command hello-to:

$ curl -k -u '2:asdfasdf' https://api.picloud.com/r/unique_id/hello-to/
{
    "output_encoding": "raw",
    "signature": "echo hello, {name}",
    "description": "command invoked in shell",
    "uri": "https://api.picloud.com/r/unique_id/hello-to",
    "label": "hello-to"}

Note

This information is identical to that returned by cloud.rest.info() or picloud rest info.

Executing Functions

Make an HTTP POST request to the returned URL to invoke the function on PiCloud. Keyword arguments to the function should be JSON-encoded and passed as GET or POST parameters:

$ curl -k -u 'key:secret_key' https://api.picloud.com/r/unique_id/square_func/ -X POST -d "x=5"
{"jid": 58}

This API call returns the jid of the started job which uniquely identifies the job on PiCloud, and can be used to retrieve information, kill, delete or get the job’s result, through either cloud or the REST API.

$ curl -k -u '2:asdfasdf' https://api.picloud.com/r/2/hello-to/ -X post -d name=\"einstein\"
{"jid": 59}

Note

If you are familiar with webhooks, note that the semantics of invoking the published function is the same as calling a webhook.

Warning

As arguments must be specified by keyword, there is no way to specify a variadic function’s unnamed arguments (*args). A variadic function can only accept keyword arguments (**kwargs).

JSON limitations

Both the arguments and result, if any, of the function must be encodable by JSON. The data structures JSON supports is limited compared to the Python pickler; see the docs for more information. If you cannot represent your arguments/return values in JSON, you will need to use binary encoding.

Job Management

Much of the job management functions in cloud have REST analogs. All of the below functions, unless otherwise noted, can be used regardless of whether a job was created through the REST API or cloud.call().

Information

The REST version of cloud.info() is GET https://api.picloud.com/job/?jids=your_id:

$ curl -k -u 'key:secret_key' https://api.picloud.com/job/?jids=58
{
  "info":
  {
    "58":
    {
      "status": "done",
      "exception": null,
      "runtime": 0.1,
      "stderr": "",
      "stdout": "Squaring 5"
     }
  }
}

You can limit the data returned by specifying specific fields in the GET args:

$ curl -k -u 'key:secret_key' https://api.picloud.com/job/?jids=58&field=status&field=stdout
{
  "info":
  {
    "58":
    {
      "status": "done",
      "stdout": "Squaring 5"
     }
  }
}

Killing

cloud.kill() can be invoked by a POST to https://api.picloud.com/job/kill/?jids=your_id:

$ curl -k -u 'key:secret_key' -X POST https://api.picloud.com/job/kill/?jids=58

If the jids field is not provided, all of your jobs will be killed.

Deleting

cloud.delete() can be invoked by a POST to https://api.picloud.com/job/delete/?jids=your_id:

$ curl -k -u 'key:secret_key' -X POST https://api.picloud.com/job/delete/?jids=58

Batch Queries

All of the above queries can deal with multiple jids.

  • And: jids=1,4 specifies jobs 1 and 4
  • Range: jids=1-4 specifies jobs 1 through 4 exclusive (jobs 1, 2, 3).
  • Combination: jids=1-4,5-8 specifies jobs 1, 2, 3, 5, 6, and 7.

Example:

$ curl -k -u 'key:secret_key' https://api.picloud.com/job/?jids=1-4,5-8

Obtaining Results

After verifying that your jobs status is ‘done’ with cloud.info (GET https://api.picloud.com/job/?jids=your_id&field=status), you can obtain its result.

cloud.result() is emulated by GET https://api.picloud.com/job/result/?jid=your_id. (Note the singular jid). This function returns a JSON-encoded result:

$ curl -k -u 'key:secret_key' https://api.picloud.com/job/result/?jid=58
{
  "result": 25
}

If the job is not yet available or has errored, you’ll receive an error such as:

{
  "error":
  {
    "code": 455,
    "msg": "Requested job is not done.",
    "retry": false
  }
}

Note

The result can be retrieved only for one job per HTTP request; hence jid not jids.

Warning

Only jobs that were created with the PiCloud REST API can have their result accessed through REST.

If your job has errored, you can obtain exception information with cloud.info, specifically GET https://api.picloud.com/job/?jids=your_id&field=exception:

$ curl -k -u 'key:secret_key' https://api.picloud.com/job/result/?jid=13
{
  "info":
  {
    "13":
    {
      {
        "exception": "Traceback (most recent call last):\n  File \"test.py\", line 21, in <lambda>\n  File \"test.py\", line 15, in throwError\nNameError: This is an error\n"}}
      }

     }
  }
}

Binary Data

If arguments or return value cannot be represented with JSON, you will need to make use of the the REST API’s binary mode. Binary data is represented in python as an str object. You are responsible for decoding binary arguments and encoding binary results. Alternatively, you can use Bucket to transfer binary objects.

Imagine we have a simple function to create a thumbnail from a JPEG image, similar to the example shown earlier:

def downscale_image(img, size = (32, 24)):
  """Reduce image to a fixed size"""
  import Image
  img.thumbnail(size, Image.ANTIALIAS) #thumbnail image
  return img

Unfortunately, a PIL Image object cannot be encoded in JSON, so we will need to decode an Image from raw JPEG data (an in-memory binary string) and encode it into JPEG data (again, a binary string):

def downscale_image_wrapper(img_str, size = (32, 24)):
    import Image
    img = Image.open(StringIO(img_str))
    downscaled_img = downscale_image(img, size = size)
    out_file = StringIO()
    downscaled_img.save(out_file, 'JPEG')
    return out_file.getvalue()

Binary Arguments

Binary arguments are passed in as multipart/form-data as a file upload (meaning the MIME Content-Disposition sub-header has a filename parameter):

$ curl -k -u 'key:secret_key' https://api.staging.picloud.com/r/3/downscale_image_wrapper -F img_str=@something.jpg
{ "jid": 15029 }

(where something.jpg is a JPEG image on disk)

In Python, you can make use of a third-party package, Poster, to upload multipart/form-data. You must specify a filename for an entry to be recognized by PiCloud as binary data:

params = []
# ....
params.append( MultipartParam(
"image",
filename='random_filename', #This field must be set!
filetype='application/octet-stream',
value=some_binary_data) )

On Google App Engine, you can use urlfetch with Poster as follows:

datagen, headers = poster.encode.multipart_encode(params)

data = str().join(datagen)

result = urlfetch.fetch(
     url=your_url,
     payload=data,
     method=urlfetch.POST,
     headers=http_headers)

Note

You can intermix binary and JSON arguments. JSON arguments appear in the multipart/form-data as text fields (no filename set in Content-Disposition / Poster MultipartParam).

Binary Results

If your result is binary, you will need to set out_encoding in cloud.rest.publish() to “raw”. In “raw” mode, the REST API result function (https://api.picloud.com/job/result/)’s will respond with the requested job’s result formatted as an octet stream (string). No JSON header will exist in the response. Of course, this mode requires that the published function’s result be a string.

If you are unsure if a result of an arbitrary job will be binary or not, you can check the Content-Type header in the response. JSON responses will have Content-Type: application/json set while raw responses will have Content-Type: application/octet-stream set.

Example:

cloud.rest.publish(downscale_image_wrapper, label='downscale_image_wrapper', out_encoding = 'raw')
# Access return value of an invocation of downscale_image_wrapper
$ curl -k -u 'key:secret_key'  https://api.picloud.com/job/result/?jid=15029 > out_img.jpg

# You can now open out_img.jpg with a standard image viewer

Bucket

You can use the REST API to access your Bucket.

Get

The REST equivalent of cloud.bucket.get() is a two step process. First:

$ curl -k -u 'key:secret_key' https://api.picloud.com/bucket/get/ -d name=your_file_name

This returns a JSON object:

{
  "ticket": {
    "Date": "Wed, 24 Aug 2011 00:11:18 +0000",
    "Authorization": "AWS AKIAJCY7JV52WD4MJSNQ:sfs4g7Dy6GkNnwq0PFW19kQIz4E="
   },
    "params": {
      "action": "https://pi-user-buckets.s3.amazonaws.com/bJjdahprLHkuwGkqnsnEdtMepiAEYmGA0qL4RFyONx/q.py",
      "bucket": "pi-user-buckets",
      "key": "bJjdahprLHkuwGkqnsnEdtMepiAEYmGA0qL4RFyONx/q.py",
      "size": 1728}
    }
}

To retrieve the file, issue a GET request to the URI in params[‘action’] field and copy all fields from the ticket dictionary to the HTTP request headers:

$ curl -H Authorization:"AWS AKIAJCY7JV52WD4MJSNQ:sfs4g7Dy6GkNnwq0PFW19kQIz4E=" -H Date:"Wed, 24 Aug 2011 00:11:18 +0000",  https://pi-user-buckets.s3.amazonaws.com/bJjdahprLHkuwGkqnsnEdtMepiAEYmGA0qL4RFyONx/q.p

The result will be an octet stream whose contents are those of the file requested.

Warning

The ticket you receive from PiCloud can only be used once and must be used within 10 minutes.

Put

The REST equivalent of cloud.bucket.put() is similar. First POST a request to make a new file to PiCloud:

$ curl -k -u 'key:secret_key' -X POST https://api.picloud.com/bucket/new/ -d name=your_file_name

This returns a JSON object similar to the Get example:

{
  "ticket": {
    "AWSAccessKeyId": "AKIAJCY7JV52WD4MJSNQ",
    "success_action_redirect": "https://api.picloud.com/bucket/uploaded/3/13/63634b0d30d23061fa573b7f97677536182f6776/",
    "acl": "private",
    "key": "c79038dc470cd016e51bae8f7243ebcee8508971/your_file_name",
    "signature": "5OnebYdDRtBbhiwg7rSoOLsipBQ=",
    "policy": "eyJjb25kaXRpb25zIjogW3siYnVja2V0IjogInBpLXVzZXItZmlsZXMtdGVzdCJ9LCB7ImtleSI6ICJjNzkwMzhkYzQ3MGNkMDE2ZTUxYmFlOGY3MjQzZWJjZWU4NTA4OTcxL3RyZWVfbmV3In0sIHsiYWNsIjogInByaXZhdGUifSwgeyJzdWNjZXNzX2FjdGlvbl9yZWRpcmVjdCI6ICJodHRwOi8vc3RhZ2luZy5waWNsb3VkLmNvbS9weWFwaS9maWxlL3VwbG9hZGVkLzMvMTAzLzYzNjM0YjBkMzA1MjMwNjFmYTU3M2I3Zjk3Njc3NTM2MTgyZjY3NzYvIn0sIFsic3RhcnRzLXdpdGgiLCAiJENvbnRlbnQtVHlwZSIsICIiXV0sICJleHBpcmF0aW9uIjogIjIwMTEtMDgtMjRUMTM6NTY6MzZaIn0=",
    "Content-Type": "application/octet-stream"
  },
  "params": {
    "action": "https://pi-user-buckets.s3-external-1.amazonaws.com/"
  }
}

To put the object you must POST to URL in params[‘action’] with multipart/form-data. All items in ticket become form data. In addition, the actual file contents will be provided as a file upload with key ‘file’. Curl example:

$ curl -L "https:/.s3.amazonaws.com/" -F "AWSAccessKeyId=AKIAJCY7JV52WD4MJSNQ"  -F "success_action_redirect=https://api.picloud.com/bucket/uploaded/3/106/9f19215d18200977284e4cc506a04bde70c030fb/" -F "acl=private" -F "key=c79038dc470cd016e51bae8f7243ebcee8508971/tree_new"  -F "signature=Pw+QlFMNwsJZ5joo4NBDaYYxAQs=" -F "policy=eyJjb25kaXRpb25zIjogW3siYnVja2V0IjogInBpLXVzZXItZmlsZXMtdGVzdCJ9LCB7ImtleSI6ICJjNzkwMzhkYzQ3MGNkMDE2ZTUxYmFlOGY3MjQzZWJjZWU4NTA4OTcxL3RyZWVfbmV3In0sIHsiYWNsIjogInByaXZhdGUifSwgeyJzdWNjZXNzX2FjdGlvbl9yZWRpcmVjdCI6ICJodHRwOi8vc3RhZ2luZy5waWNsb3VkLmNvbS9weWFwaS9maWxlL3VwbG9hZGVkLzMvMTA2LzlmMTkyMTVkMTgyMDA5NzcyODRlNGNjNTA2YTA0YmRlNzBjMDMwZmIvIn0sIFsic3RhcnRzLXdpdGgiLCAiJENvbnRlbnQtVHlwZSIsICIiXV0sICJleHBpcmF0aW9uIjogIjIwMTEtMDgtMjRUMTQ6MTU6NDhaIn0=" -F content-type="application/octet-stream" -F file=your_file_to_upload

Warning

You must obey the HTTP 303 redirect (curl -L option) that this query will return (that is to do a GET to the redirect url) or PiCloud will not know that your bucket object is ready! (urllib2 and urlfetch do this automatically).

Other functions

cloud.bucket.list():

$ curl -k -u 'key:secret_key' https://api.picloud.com/bucket/list/
{"files": ["file_you_uploaded", "another_file"].
  "truncated": false}

cloud.bucket.exists():

$ curl -k -u 'key:secret_key' https://api.picloud.com/bucket/exists/ -d name=your_file_name
{"files": ["file_you_uploaded", "another_file"] }

cloud.bucket.remove():

$ curl -k -u 'key:secret_key' https://api.picloud.com/bucket/exists/ -X POST -d name=your_file_name
{"deleted": true }

Queue

Queues can be manipulated via REST. Please be sure to read the Queue documentation first, since this page only documents the url endpoints to use.

There is no need to explicitly create queues. Any operation on a queue with a given name (in the below examples your_queue_name) implicitly “creates” a queue with that name.

Number of Items

GET https://api.picloud.com/queue/your_queue_name/count/

$ curl -k -u 'key:secret_key' https://api.picloud.com/queue/queue-1/count/
{"count": 0}

General Information

GET https://api.picloud.com/queue/your_queue_name/

Note that ‘count’ is included.

$ curl -k -u 'key:secret_key' https://api.picloud.com/queue/queue-1/
{"info":
  {"func_name": "__main__.some_attachment.py:14",
  "queued_jobs": 0,
  "count": 0,
  "name": "etsy-input",
  "created": "2013-03-05 23:26:56",
  "processing_jobs": 0,
  "output_queues": ["queue-2"],
  "max_parallel_jobs": 2,
  "has_attachment": true,
  "readers_per_job": 5}
}

Message Handling

Message Encoding

A queue message will be wrapped in a JSON dictionary that is less than 64 KB in length. All dictionaries include a “datatype” which indicates how a message is formatted. Possible datatypes are shown below.

JSON

In the simplest case, the datatype is “JSON”, meaning that the message is itself JSON-encoded. (Note that you need to pass both the message wrapper and message itself through JSON encoding!)

{"datatype": "json", "message": "6"}

encodes the integer 6 and

{"datatype": "json", "message": "\"6\""}

encodes the string 6 (note the escaped quotes).

Pickle

If a message was pushed from Python and could not be encoded into JSON (e.g. it is a python instance), the message is first serialized with the python pickler and then base64 encoded. e.g.:

{"datatype": "base64_python_pickle", "message": "gAJjX19idWlsdGluX18Kb2JqZWN0CnEBKYFxAi4="}
References

If a message is larger than 64KB, it cannot be saved directly. Rather a reference to a bucket object may be made:

{"datatype": "json", "redirect": "bucket", "key": "queue/queue-1/cc2b363b-a286-4a6c-9a4e-29bf34324acb"}

If “redirect” is set to bucket, the actual message will be located at the bucket object specified by key (e.g. cloud bucket get key). “datatype” indicates the datatype of the data stored in the bucket; it will be either “json” (json data) or “python_pickle” (data serialized with the python pickler; note that no base64 encoding is needed for bucket).

The python implementation of cloud.queue.CloudQueue.push() and cloud.queue.CloudQueue.pop() automatically handles references. When a reference is popped with cloud.queue.CloudQueue.pop(), the object is automatically deleted from bucket storage.

Push

POST to https://api.picloud.com/queue/your_queue_name/push/

Set one or more message post parameters to an encoded message. You can set the delay post parameter. Delay works as described in cloud.queue.CloudQueue.push().

example:

$ curl -X POST -k -u 'key:secret_key' -d message='{"datatype":"json", "message" : "6"}' https://api.picloud.com/queue/queue-1/push/

This creates a single message with an integer of “6”.

You can push up to 800 messages in a single request.

Pop

POST to https://api.picloud.com/queue/your_queue_name/pop/

You can set the timeout post parameter as well as the max_count post parameter. Both work as described in cloud.queue.CloudQueue.pop().

A JSON dictionary of encoded messages will be returned.

Example:

curl -d timeout=1 -X POST -k -u 'key:secretkey' https://api.picloud.com/queue/queue-1/pop/
{
"messages" : [
{"datatype": "json", "message": "6"},
{"datatype": "json", "message": "7"},
]}

Pop with Tickets

To perform a pop that requires each message be acknowledged (for robustness), POST to https://api.picloud.com/queue/your_queue_name/pop_tickets/. Each message in the JSON dictionary will include a “ticket” key.

e.g.

curl -d timeout=1 -X POST -k -u 'key:secretkey' https://api.picloud.com/queue/queue-1/pop_tickets/
{
"messages" : [
{"datatype": "json", "message": "6", "ticket" : "qeu239784dkjs"},
]}

To acknowledge the message, POST one or more of the tickets returned by pop_tickets with key “ticket” to https://api.picloud.com/queue/your_queue_name/ack/ e.g.:

$ curl -X POST -k -u 'key:secret_key' -d ticket='qeu239784dkjs' https://api.picloud.com/queue/queue-1/ack/

To update the deadline, POST one or more ticket keys as POST data, along with a new deadline (POST data - in seconds) to https://api.picloud.com/queue/your_queue_name/update_deadline/

Attachments

It is currently only possible to attach a Python function to a queue. If you would like beta access to the queue rest shell interface, please file a support ticket.

Other functions

cloud.queue.CloudQueue.detach():

$ curl -X POST -k -u 'key:secret_key' https://api.picloud.com/queue/your_queue_name/detach/

cloud.queue.CloudQueue.delete():

$ curl -X POST -k -u 'key:secret_key' https://api.picloud.com/queue/your_queue_name/delete/

cloud.queue.list():

$ curl -k -u 'key:secret_key' https://api.picloud.com/queue/
{"queues": ["queue-1", "queue-2"] }

REST from JavaScript

By default, PiCloud will respect the same origin policy and not allow AJAX requests to the REST API. This policy is in effect due to two large security concerns:

  1. All REST requests require the api key and api secret key. Making requests from JavaScript require exposing such keys to the browser’s user.
  2. If your Basic Authentication credentials become cached, malicious websites could manipulate your PiCloud account using your cached credentials.

If you are building a web service that uses PiCloud, we highly recommend proxying all requests to PiCloud through your server. If you do wish to nonetheless make calls from JavaScript, and understand the security concerns, file a support ticket and we can white-list a specific API key to support Cross-origin resource sharing.