Storage as a Service Extension Overview

In previous articles we have looked at what an API extension is and how they work internally in context of Storage as a Service.  Let's look at the implementation of an API extension from a business logic standpoint that delivers our STaaS service using vCloud Director API extensions. 

Storage Extension Process


If we look at the 7 steps in our overall API process, steps 1,2,6 and 7 have been covered in the previous articles.  Steps 3, 4 and 5 have been implemented as a Java process that subscribes to our AMQP queue in order to receive API requests, and process them.  The Java process starts a new thread for each extension service registered with the extension server that processes the incoming API messages.

The AMQP Server object, which is the thread that will process the API messages subscribes to the AMQP queue that is registered with the service extension, as it was defined in the service extension XML definition.  (See introduction to API extensions for details).  The AMQP server thread takes the AMQP message and deserializes the message body into 3 objects, using the Jackson JSON libraries.  The deserialization results in 3 objects, HttpMessage, SecurityContext, and Context.  These 3 objects are then submitted to the HttpExtensionService object for processing.  The HttpExtensionService class is the class which our API extension business logic needs to implement.  In this example, our HttpExtensionService has been implemented as a class called StorageService.  StorageService receives the 3 objects from AmqpServer and then processes the API request, implementing the business logic.

StorageService has generic methods for interfacing with the underlying storage subsystem.  StorageService takes the HttpMessage and consumes the POST message body which is an XML doc with the storage request parameters.  These are the details of the storage request, such as storage size, IQNs to map to the new Lun and storage state (online, offline).  StorageService processes the XML and then calls the vendor specific implementation of how to complete the tasks of the generic methods.

In this example we have a class called OntapController, which takes the reqest to create/destroy/online/offline storage from the StorageService object and actually carries it out on the NetApp 3240 we used for this example.  StorageService then returns an HttpReply object back to the AmqpServer with the results of the provisioning task, which are then serialized into JSON and sent to the reply exchange for consumption by the vCloud Director cell.

I'll post the source code for this example with usage instructions in a follow up post.


VMworld Barcelona Sessions

I had a great time presenting a number of sessions at VMworld in San Francisco this year and , I will be fortunate enough to be delivering a number of sessions at VMworld Barcelona this year.

GD41: vCloud Director Architecture, IntegraIon and Orchestration  (11am)
INF-VSP2033 : Auto Scaling and Cloud Bursting in the Hybrid IaaS Cloud  (5pm)

Experts05:  Meet the Experts (11am)
OPS-CSM1379:  Extending vCloud Director (2pm)
OPS-CSM2248:  A Class on Deploying a Production Cloud Architecture (5pm)

Experts12:  Meet the Experts (12pm)
GD41:  vCloud Director Architecture, IntegraIon and Orchestration (3pm)

Anyone who read my earlier articles on vCloud API extensions and wants to find out more, if you're going to be at VMworld the session OPS-CSM1379 is all about API extensions.  It is a very technical session, no marketing or product discussions of any time.  It is all about building API extensions.  We're going to go through the process of building an extension, how they work and demonstrate a few example extensions we created for the session.  Specifically, we created an extension that provides Storage as a Service, exposes FT to vApps in vCloud Director, and simple database as a service.

Session OPS-CSM2248 is a little misleading in the title and description unfortunately, but bascially we take the audience through architecting a cloud environment based on requirements and Q&A from them interactively and in real time.  If you have any questions that you'd like to see whiteboarded out on the spot related to building a vCloud environment, this is a must attend session.  I'll be delivering the session with my colleague Thomas Kraus.  Here's the session in action at VMworld San Francisco.  http://www.vmworld.com/docs/DOC-6363  You'll need to sign up for a vmworld.com account if you don't already have one.  (It's free)


Registering API Service Links

Where are my API extension REST methods?

So far we have looked at how API extensions work, how to register a new service extension, and what the messages look like that are sent from the API call to the AMQP exchange for our business logic to process.  However, if we were to go and do an http GET on an organization within our vCloud environment, we would not see anything special about that organization.  There would be no additional API calls (URLs) returned by the GET that would indicate we in fact have an API extension available to us that we can call against that organization.

What we can do is register custom links against entities within vCloud Director (vCD) environment that will be returned whenever we perform an http GET against that entity within vCD.  In this example we are registering against the organization entity type (application/vnd.vmware.vcloud.org+xml) but we can register links against any entity type we wish to add service links to.

Before we register our links, lets look at what an http GET looks like against our organization without any new API service links

Cloud wide vs entity specific links

When we register a service link with vCD, we can register the links with the entity type, which will add the links to all entities within our vCloud, or we can register them with specific URNs for the given entity type.  In simple terms, in context of our example we can add our links to all organizations within our vCloud, or to specific organizations within our vCloud by specifying the URNs of the organizations we want to register the links against in the XML that defines the links.

Defining a service link

In order to define our service link(s) we create an XML definition of the links we wish to register and post it to the /admin/extension/service vCloud REST method the exact same way we registered our service extension itself.  Nothing is different from the API interaction standpoint.  Only the XML is different.

If we look at a service link we want to register against our organization entity within vCD, we can from that easily define the XML that we require.

Desired link


Service link XML definition


Breaking down the components of this XML snippet

{baseUri} will be replaced with the address of the vCD environment you're connecting to, or in our example case it translates to
{resourceId} will be replaced with the URN of the entity we are making the API call against, or in this case the URN of the organization we perform the API call against.

We need to specify more than simply the desired link.  vCD needs to know what entity type to register the link against, and optionally what URNs to register it against if it is not a vCloud wide link being registered.  This is an example of the ServiceLinks section of an API extension being registered to provide all of the calls we want for our external iSCSI as a Service implementation, to be registered against all the organizations within our vCloud.  We could have optionally specified the resourceId of the resource we wanted the links registered against.



Now after we register our service links against our organization, lets take a look at what an http GET against our organization looks like.

As you can see we now have new links we can call against our organization in order to interact with our storage as a service API extension.  This allows us to have whatever code (or person) is interacting with the API to automatically discover what methods are exposed by the organization for interacting with it.  We don't have to have prior knowledge of the methods to call, they are exposed by the organization.

Receiving and consuming API extension messages

In the previous articles, we looked at the underpinnings of the vCloud API extension framework, and how to register an extension.  We registered an extension for delivering iSCSI storage as a Service.  Now lets look at what happens when we call our new API extension.  In this example, we will call the following API method.


Note that the organization id (1d413961-a514-49ee-b62a-d782f43eab11) could be any ID, as this is generated by vCloud Director (vCD) to and is unique to the organization resource.

When we call our new API URL, an AMQP message is sent to the specified exchange (extstorage), with the routing key set (extstorageiscsi).

AMQP Message

When vCD sends the API call as a message to the exchange it sets the following message headers.

AMQP Headers

correlationId:  vCD sets unique correlationId in every message sent to extension and the extension must set the same value in the corresponding response
reply-to:  The extension must use this value as routingKey when sending the response back

Custom AMQP headers added by vCD

messageType:  Either 'ProcessHttpRequest' or 'ProcessHttpResponse' and indicates whether request or response is being forwarded
replyToExchange: The exchange which the extension should publish its response to

Note that for API extensions, the exchange the reply message from the API business logic must be sent to is different from the exchange that the business logic picks up the initial API call message from.

AMQP Message Body

The message body of the API message consists of a JSON array containing 3 objects.  HttpMessage, HttpSecurityContext, Context.  You can deserialize this using Jackson or another JSON processing library, or if you are writing the message handling/business logic of your API extension in Javascript such as with vCenter Orchestrator, you can perform an eval() on the JSON string to return the objects as an Array object.  The objects are POJOs as follows:

HttpMessage Object

private boolean isRequest;
private String id;
private String method;
private String requestUri;
private String queryString;
private String protocol;
private String scheme;
private String remoteAddr;
private int remotePort;
private String localAddr;
private int localPort;
private Map<String, String> headers;
private byte[] body;
private int statusCode;

 Note that when your API extension is sending its reply message back to vCD via the reply exchange, if you do not want the vCD REST Handler to process the message your extension generates, you must set isRequest to false.  Otherwise, your reply message will be handled by the next priority in line API handlers (registered extensions that match on regex, or vCD's REST handler) and things can and likely will go sideways, unless having the message be processed as a modified request is your intended design.

HttpSecurityContext Object

private String user;
private String org;
private List rights;
private Map<String, Object> parameters;

For the HttpSecurityContext object, the user and org are the URN of the user and organization the user belongs to making the API call.  You can use the search facilities of vCloud Director to convert the URN into the entity (user, org).  The rights list contains all of the URNs of the vCloud rights assigned to the user making the API call.

The following is a sample JSON message body from an AMQP message sent by an API call to the following URL as an HTTP GET.  Note that if we did a POST to the API URL, the method field would be set to POST, and we would have another field called body, which would be a Base64 encoded byte array of the content sent as part of the HTTP POST.

    "headers":{"Cookie":"...", "User-Agent":"...", ...},

If you are using Jackson to deserialize the JSON you do not have to do the Base64 decoding.  If you are using eval() in vCenter Orchestrator or another solution, you will have to do the Base64 decode on the body you receive.  You also will have to Base64 encode when you send your reply message.

The business logic of your API extension should in its simplest form consume the following information from an http GET to the API URL, and perform its desired operations

  • method
  • requestUri

Optionally, if you are performing an http POST operation, consume the byte array from the HttpMessage body, and use that content to drive the business logic of the new API extension.

The following code is a complete message consumption loop implemented in Java.  Note that this method calls on a few objects that are not referenced in this code, specifically

  • connection:  The connection parameters to the AMQP exchange
  • amqpQueue: name of the queue to pull messages from
  • mapper:  Jackson ObjectMapper
  • extensionService: An object that implements the functionality of the extension itself.  It has two methods, processHttpRequest() and processHttpResponse(), both return an HttpMessage object that can be serialized and sent back to vCD via the replyToExchange.
    public void retrieveMessages() throws IOException, InterruptedException {
        Channel channel = connection.createChannel();
        QueueingConsumer consumer = new QueueingConsumer(channel);
        channel.basicConsume(amqpQueue, true, consumer);
        while (true) {
            QueueingConsumer.Delivery delivery = consumer.nextDelivery();
            BasicProperties props = delivery.getProperties();
            Map<String, Object> headers = props.getHeaders();
            String msgType = headers.get(MSG_TYPE_HEADER).toString();
            String replyToExchange = headers.get(REPLY_EXCHANGE_HEADER).toString();

            byte[] body = delivery.getBody();
            BasicProperties replyProps = MessageProperties.PERSISTENT_BASIC;

            HttpMessage result = null;
            if (msgType.equals(REQUEST_TYPE)) {
                Object[] args = mapper.readValue(body, Object[].class);
                HttpMessage request = mapper.convertValue(args[0], HttpMessage.class);
                HttpSecurityContext securityContext = mapper.convertValue(args[1], HttpSecurityContext.class);
                Map<String, Object> context = mapper.convertValue(args[2], Map.class);
                result = extensionService.processHttpRequest(request, securityContext, context);
            } else if (msgType.equals(RESPONSE_TYPE)) {
                Object[] args = mapper.readValue(body, Object[].class);
                HttpMessage response = mapper.convertValue(args[0], HttpMessage.class);
                result = extensionService.processHttpResponse(response);
            } else {
                throw new RuntimeException("Unknown message type: " + msgType);

            byte[] replyMsg = mapper.writeValueAsBytes(result);
            channel.basicPublish(replyToExchange, props.getReplyTo(), replyProps, replyMsg);

In the next article we will look at how we can register service links for our new API extension in order to provide visibility to the new methods it supports when performing a GET operation on the entity we register the links with.  In this example, we will register links against the Organization entity, as that is what our service regex is looking for.  At this point you should have an idea of how the API extension works, how to register one and how to interact with it using the RabbitMQ Java client library.

Joomla templates by a4joomla