Developer Blog

MD5 Validation for Perl

2010-01-13T19:55:00Z

Verifying the files you have stored on our servers is an important part of data integrity. To calculate the MD5 we store on our servers in Perl the quick solution would be to use the built in b64digest output which is part of the Digest::MD5 class.

For the test file the MD5 being returned by Nirvanix ListFolder is: bNIkeoRicdpTMoPyuN/4rg==


use Digest::MD5;
use MIME::Base64;
    my $file = shift || "./somefile.txt";
    open(FILE, $file) or die "Can't open '$file': $!";
    binmode(FILE);
    print Digest::MD5->new->addfile(*FILE)->b64digest, " $file\n";

output:
bNIkeoRicdpTMoPyuN/4rg ./somefile.txt

For some reason this does not produce the expecting padding of: ==

If you change the script slightly to call the base 64 encoding externally using the digest output (binary MD5) then it will report the correct value.

use Digest::MD5;
use MIME::Base64;
    my $file = shift || "./somefile.txt";
    open(FILE, $file) or die "Can't open '$file': $!";
    binmode(FILE);
    print encode_base64(Digest::MD5->new->addfile(*FILE)->digest), " $file\n";

output:
bNIkeoRicdpTMoPyuN/4rg==
./somefile.txt

It is important that you do not just add == to the results of the b64digest since this is not always returned (though usually is). There are cases when the extra bytes are used by the base64 algorithm.

SDN URL Redirection

2009-09-16T16:41:00Z

The dynamic nature of our system allows us to automatically route uploads and downloads to the best node for optimal performance and to ensure system availability during maintenance or heavy load. This is done by coding URLs to “http://services.nirvanix.com” and allowing the SDN to automatically redirect them to the appropriate node. This is a very important feature that all customers need to follow to ensure the best user experience.

SDN files can be accessed in three ways, specifically by using a: (1) session-based URL, (2) hosted-item URL, or (3) tokenized URL.

1. A session-based URL is codified using the form: “http://services.nirvanix.com/SESSIONTOKEN/PATH/FILENAME”. The SESSIONTOKEN is created by the “Login” API call. This URL is automatically redirected to the closest active node that contains the requested SDN file.

2. A hosted-file URL is codified using the form: “http://services.nirvanix.com/APPNAME/ACCOUNT/PATH/FILENAME”. Hosted files are created using the “CreateHostedItem” API call. Similar to session-based URLs, this URL is automatically redirected to the closest active node that contains the requested SDN file.

3. A tokenized URL for an SDN file is generated by the “GetOptimalURLs” API call. The resulting URL points to a specific node that contains the file when “GetOptimalURLs” was called. However, if the node is unavailable or the SDN file is subsequently migrated to a different node, then the tokenized URL will return an error and not resolve to the file (even if the URL hasn’t expired yet). For that reason, we recommend specifying a short expiration period for tokenized URLs and implementing error handling, e.g. regenerating the URL using “GetOptimalURLs” if the file is unavailable.

All three of these methods use IP-based geo-location services to determine the closest active node. Session-based and hosted-item URLs resolve to the datacenter closest to the client accessing the file. Tokenized URLs are fixed to the node closest to the IP address passed to the “GetOptimalURLs” API call.

Uploads are treated similarly to the “GetOptimalURLs” call, such that a token is generated that allows you to upload to a specific node. You must call “GetStorageNode” or “GetStorageNodeExtended” for each file being uploaded.

A Feature Review

2009-06-20T01:09:00Z

I wanted to go over the different releases over the last year or so to give a quick overview of what has been done. It can be a challenge to read through our release notes in the forums if you are not already a customer so a roll-up is in order.

The build out of 4 nodes including Japan and Europe.

CloudNAS for Windows and Linux - We have been focusing on offering significant speed and simplicity to allow you to mount the file system as a network drive. We have released many new features allowing significant performance improvements and compatibility with many different backup and other server solutions.

Improved services server - The services servers received a significant improvement to speed and scalability to allow for even faster requests and allowing significant redundancy.

Output as JSON - We announced output as JSON in our latest release which allows integration with JavaScript making web site integration easier.

Network Improvements - All of our nodes are continually being improved for network performance. The overall transfer speed for many of our customers have been many fold improvements for upload and download. This is something we consider one of the most important features of our service.

Reliability - While I cannot detail the improvements made in this regard there have been significant changes like improved error handling in the last 6 months. Features cannot be measured if there is no service running so this is a top concern for us.

White Lists - The ability to create IP ranges that restrict download of public files to only those IPs.

Customer Support - The management portal has been enhanced in numerous ways including a ticketing system for support entries and full child account modification and creation.

This is not a complete list but it is a quick way to catch up on what we have been working on in our drive to add features and provide a great platform to build your application on.

Streaming and Seeking in Silverlight with Nirvanix

2009-05-08T17:36:00Z

Silverlight offers some great video playback features in terms of allowing progressive download and offering a method to index and seek WMV. This is a great way to allow complete freedom in displaying movie playback while using a progressive download server such as a Nirvanix account.

The normal buffering that you would see with any progressive downloaded Flash or Silverlight video is still happening, but because the WMV is indexed the player can skip to any point in the video and start buffering / playing from that point. Once you have your video completed you can upload the file along with some HTML to invoke the player. The player I have chosen below is quite nice because it is open source, supports the seeking capability and is quite attractive.

The link below is a demo of movie length content which lets you move the slider to any point.

Demo:
http://services.nirvanix.com/NirvanixDrive/barrynirvanix/silverlight/silverlighttest.html

    <object data="data:application/x-silverlight-2," type="application/x-silverlight-2" height="480" width="640">
        <param name="source" value="VideoPlayer.xap">
        <param name="background" value="white">
        <param name="initParams" value="m=http://services.nirvanix.com/App/User/PathtoFile/Video.wmv">
               <param name="minruntimeversion" value="2.0.31005.0">
        <a href="http://go.microsoft.com/fwlink/?LinkId=124807" style="text-decoration: none;">
             <img src="http://go.microsoft.com/fwlink/?LinkId=108181" alt="Get Microsoft Silverlight" style="border-style: none;">
        </a>
    </object>

The object above can be placed in your HTML and will render the video using the VideoPlayer.xap. You can place the html, player and video in a directory and share that directory to allow the file to be played back.

The HTML code used to load the player can be downloaded here:

The SL2 Video Player:
http://sl2videoplayer.codeplex.com/

Video Playback Tutorial

2008-12-16T18:48:00Z

Often you will want to play the video stored on your Nirvanix account. One of the easiest formats to play back would be FLV since Adobe has done most of the hard work in flash already. You have the option to either create a player in flash or you can purchase one of the many that are available. This tutorial will describe how to convert a video in avi format to FLV then upload, transcode and setup a player.

Uploading the AVI

For this step we will use the PHP Web Client. You can find a sample site running at: http://www.nirvanixtest.com/webclient/

If you intend to use this extensively you should get the source and put the site up on your own server. To upload we will create a folder in the root called "video"

Transcoding to FLV

After the upload is complete you can browse to the upload folder and click on the file you want to transcode and click on Transcode Video.

Once you have selected the video to transcode you will need to type the destination file and path along with the transcode quality.

After you click on "Transcode File" It may take a few minutes to process the request in the background. Depending on the size of the file the time will vary. You can refresh the destination folder to know when the conversion has completed. Normally you would have a process in the background listening to the callbackURL which will be called when the video transcode is complete.

Embed Flash in HTML

Once you have the video converted to FLV you will need to upload a player to this directory. For this tutorial we have selected the moxiecode FLV player because it is a simple example that follows the logic that most web flash applications use. You can find a copy at: http://oos.moxiecode.com/flvplayer/ along side: http://blog.deconcept.com/swfobject/ which is used to render the swf object through javascript without too much more work.

Upload the flvPlayer.swf and swfobject.js to the movies folder along side the FLV. This will be the movie player you will be using that is embedded in the html page we will be creating. Once this is done you can upload the html page to the same folder.

You can view the working html sample here: http://services.nirvanix.com/Movie Viewer/movietest/movies/video.html

Silverlight Support

2008-09-16T17:44:00Z

We are happy to announce we have added the ability to access your content from Silverlight. The latest release added a file at the root of both servies and the nodes that is a security file allowing Silverlight to connect to and download from both API calls as well as files. Part of the release was to add the Microsoft suggested Mime-types for hosting Silverlight files as well. Some restrictions on hosting the Silverlight application may be having to call the node directly to avoid the redirect.

The security file can be accessed at: http://services.nirvanix.com/clientaccesspolicy.xml and at the root of each node as well.

I look forward to our Silverlight developers showing off the latest interfaces to access both API calls and file downloads in the future. Please post a comment if you have a questions or concerns.

Callbacks and communicating with processing

2008-08-14T07:50:00Z

I have received a number of direct questions from Nirvanix users regarding callbacks and thought that I would share the information I sent to them with the entire community. Our processing servers allow you to work on requests that may take a significant amount of time. This means you can transcode videos that are > 200gb or any number of other supported large file operations. When processing a large file it's necessary to move these calls to an asynchronous process to avoid keeping an active connection open. Web interfaces aren't the best mechanism for long running connections and it can take quite a while to transcode a 200gb video depending on the CODEC. This setup requires a way to let you know when we are finished with the operation, this is where callbacks are used.

When something is added to our queue it will be picked up by our processing servers and when they are done with the request they will make a call to the URL presented in the initial request. The data returned depends on what operation you were performing. There will always be a response code passed which if 0 means the command succeeded as expected. If it is non-zero there will also be an error message. This is necessary because the initial call to the method only adds the request to the queue and does not actually perform the operation. You should receive two separate callbacks, the first is a notification that the file has been added to the queue successfully (or un successfully) and the second is notification that the file has been processed successfully (or unsuccessfully). Cases where the initial notification would not be 0 include paths already existing, users reaching their set limits or other problems.

The callback URL you submit to us is in the following format http://www.mysite.com/mycallback.php which would reside on your site.

Once a file has completed processing you will receive the following information:

NVX.returnCode an integer representing the status of the upload. Did it fail? Succedd? For valid response codes: see standard return codes; return code 0 means no error.
NVX.errorMessage a string describing the error message associated with any error being returned.
NVX.absolutePath a string describing the absolute path of the file that was uploaded.
NVX.sizeBytes a long describing the size in bytes of the file that was uploaded.

Transcoding Audio / Video

The transcode calls can take some time to convert the files depending on the input and output formats as well as the quality settings. For this callback the operation will only return if the call was successful as well as any errors that have occurred during processing. If you are converting to FLV / MP3 formats you can use this callback to determine when you should make the files visible to the users of your system.

Image Resize / Rotate

The image calls usually occur very quickly and will return if the call was successful as well as any errors that have occurred during processing. Possible reasons for failure would be that the submitted format is unknown to us. Often this call is used to swap a preview image with a generated thumbnail. This can be a good queue for your system to present a resized image instead of a stock icon if you have a way to show previews of image files on your system.

Sideload

For sideloads we are actually going out on the web and making requests of external sites which can have significant overhead if the site we are contacting should be slow or unresponsive. It is critical that you capture sideload errors since this is calling to an external system which can often behave in unexpected ways. The sideload call is a great way to save external data into your account without having to proxy the files through your own system.

Upload

Uploads are different in what they return so you can do post processing on a file after it has been uploaded. Often systems using our web services allow users to upload directly to our servers which allows you to avoid proxying files and letting your customers upload the files directly to us. The challenge is having a way to update your system or local database with the files those users have uploaded. The callback can be used as a convenient way to keep your local system informed of when a user uploads their own file. Another use of this callback is to execute a secondary action when the upload is complete. You can check the file for the extension to see if its allowed, generate a thumbnail on an image or any number of operations.

Sample Callback Handler in PHP

<?php 

$myFile = "requestlog.txt";
$fh = fopen($myFile, 'a') or die("can't open file");
fwrite($fh, 'Requesting URI: ' . $_SERVER['REQUEST_URI'] . "\r\n");
fwrite($fh, 'Request Time: ' . date("D M j G:i:s T Y") . "\r\n");
fwrite($fh, 'Query String: ' . $_SERVER['QUERY_STRING'] . "\r\n\r\n");

fclose($fh);

?>

The handler will append the results of each callback to a file called requestlog.txt. Make sure your web server (PHP) has write privileges to that file.

ItemID Changes

2008-07-02T15:35:00Z

We wanted to let everyone know of a change to the ItemID structure. This is significant if you are storing the ItemID along with your file information. This ID was improved to make sure it will always be Unique across an application. Since the change will affect existing item ID's we suggest that if you are already storing the Item ID you update your database with the new value returned from GetPathInfo.

Please let us know if you have a large number of Item IDs stored or if you will have any trouble with this change. We will work with you through the transition. We will be making the change on the next release date that is targeted for next week.

Base Storage

2008-06-21T20:15:00Z

All,

We have removed the option to sign up for base storage from our sign up process. This will not affect those of you who are currently signed up for Base storage and your service activity will continue to be locked to the node you specified. We did this because of the benefits our SDN customers have experienced allowing the intelligent uploading, downloading, and movement of files throughout the global Storage Delivery Network. We continue to work hard to provide you with the highest quality Cloud Storage service possible. If you have any questions feel free to contact us at info@nirvanix.com and as always thank you for your continued support.

Staying Secure, Security Best Practices

2008-05-29T15:34:00Z

The security model we have designed allows for a few options when deciding how to stay secure. Today's topic will be to explore security and how you can be sure you know all of the primary methods for keeping data and your credentials secure.

Authentication

A session token is generated using an application key, username and password. With these three credentials you authenticate to a specific application. Its important that you secure the Application Key because like your username it is also part of the total authentication into the system.

Passwords

Each of the password change methods require a full re-authentication by the user. This is to prevent someone from stealing a session token and taking over an account. If you use SSL there should be no way for an external party to take over your account.

Session Tokens

The session token was designed to give you a sliding window in which you can operate on the APIs from a single external location. A session token is in essence the key to our web services after you have proven you are who you say you are. This token is locked to a single IP Address always. This is because a session token is something that is very specific to a user and should never be given to anyone else.

The session token not only will let you get access to the web services but it will also let you download your files. Below is a URL that has been built based on a session token that will access the file file.txt in "application1" and childuser1 which in this example is a child account.

http://services.nirvanix.com/SESSIONTOKEN/application1/childuser1/path/file.txt

Session tokens stay active for 20 minutes unless they are used. Whenever a session token is used it will be updated to be active for another 20 minutes. This means if your session token stays active it could be available indefinately. Because of this its a good idea to call logout which destroys a session token. The reason this is important is to prevent anyone who may be watching your http traffic.

The best use case for a session token is to login, do a single operation using HTTPS and then logout. However, there are a number of reasons this isn't always possible. If you have large amounts of data to pass or are extremely time sensitive the addition of SSL could be too much overhead. If you find yourself in such a situation be aware that if someone is able to get a session token they will still have to try to spoof your IP Address to use the token. This is why its always a best practice to call logout at any time you have completed your operation because that will instantly invalidate the session token assuring you that no-one else may use that token either.

Upload Tokens

The upload token is another form of validation that can be created to allow users to interact with the system. The reason the upload token is separate from session tokens is to allow the ability to pass to the end client in a web site scenario this token to do an upload. If a session token were used the end client could use that token to call any method under their account. With the addition of the GetStorageNodeExtended method you can now also lock an upload token to a specific IP and allow them to only upload to a specific directory.

Download Tokens

A token can be generated with GetOptimalURLs which lets you access a file for a specified period of time. The token can be created in a way that is similar to upload tokens that allows you to only allow a specific IP Address. The download tokens will timeout after a specified period of time. The suggestion is always to keep the timeout as short as possible since this token cannot be destroyed later. If you distribute the link and it is not locked to a specific IP Address anyone can download that file until it expires.

Password Locking

If the login system is called and fails 5 times your login will be locked. This is a temporary lock that will be released after 15 minutes. This is necessary to prevent attempts at guessing passwords. You can unlock your account at the Nirvanix Management Portal (NMP) (http://nmp.nirvanix.com/) using the security question answer.

Re-setting Application Key

The application key should be considered a secure item since it identifies your application. Should someone get your application key and you are concerned that you have been compromised you can regenerate it using the NMP. Any applications using the old key will not be able to login again until the new application key is used.

Additional Security Information

A whitepaper is available that details eavesdropping, impersonation and database attacks. You can find this and other important whitepapers at: Nirvanix Whitepapers