Developer Blog

  • Uploading Streams in Java

    The latest version of the Nirvanix Java SDK provides an interface to upload input streams directly to Nirvanix. If you are getting your data from a source that does not derive from a file this is a quick way to get the data up to Nirvanix.

    With this release we provided a StreamUploader application that demonstrates a variety of ways that stream uploads can be used.

    The first example is using a ByteArrayOutputStream to generate random data. This is purely test code but it is useful in showing how you can manipulate arrays of data then upload them to Nirvanix directly from memory.
        protected static byte[] createTestData(int length) {
            try {
                ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
    
                for (int i = 0; i 


    The code above first creates an outputStream and loops through until it has created a outputStream with enough data. The final step is to return this as a Byte Array.

    Performing the upload involves first creating a Session object which allows us to login to Nirvanix. After the session has been created the Uploader object is created and the final step is to pass our input stream to the Uploader and create an UploadSession.

                byte[] inputBuffer = createTestData(512);
                InputStream inputStream = new ByteArrayInputStream(inputBuffer);        
                long streamLen = inputStream.available();
    
    
                // Create a Nirvanix session with the credentials provided.
                Session session = new Session(username, password, appKey, appName);
    
                // Create an uploader based on the session.
                Uploader upload = new Uploader(session.getAccountLogin());
    
                // Setup upload parameters.
                upload.setIpRestricted(false);
                upload.setOverwriteExistingFiles(true);
    
                // Create an UploadSession with the stream provided.
                UploadSession uploadSession = upload.getUploadSession(inputStream, 
                        streamLen, folder, filename);
    
                UploadTransferListener listener = new UploadTransferListener(uploadSession);
    
                // Add transfer listener for progress reports.
                uploadSession.addTransferListener(listener);
    
                // Start the upload.
                uploadSession.doTransfer();
    
    
    This simple method can be done for other sources like streaming from the command line input stream as well.
  • Java SDK 1.5.1 Changes

     The changes to the Java SDK allow for significant improvements to the way uploads are performed.  The new upload implementation will allow for streams and the ability to try and control the state of your upload.  The feedback the uploader returns allows you to save the state of any upload in progress so you can continue later if you wish to temporarily pause or resume in cases of a crash.

    To start using the new uploader your code will need to change somewhat since the interface has changed and the callback mechanism has been rewritten.  Below is a sample uploader using the SDK.

     

    //~--- non-JDK imports --------------------------------------------------------
    import com.nirvanix.sdk.common.NLogger;
    import com.nirvanix.sdk.io.common.*;
    import com.nirvanix.sdk.io.upload.*;
    import com.nirvanix.sdk.session.Session;
    import com.nirvanix.sdk.transport.AccountLogin;

    //~--- JDK imports ------------------------------------------------------------
    import java.io.IOException;
    import java.util.logging.*;
    import java.util.HashMap;

    public class Upload {
        public static void main(String[] args) {
            try {
                NLogger.setLogger(createLogger("NirvanixJavaSDKSample.log"));
                uploadFile("c:\\somefile.txt", "/javaupload", false);
            } catch (Exception ex) {
                System.out.println(ex);
            }
        }
         private static void uploadFile(String localUploadFilePath, String remoteFolder, boolean useEncryption)
                throws Exception {

            String userName = "nirvanixusername";
            String password = "nirvanixpass";
            String appName = "appname";
            String appKey = "appkey";
            
            Session session = new Session(userName, password, appKey, appName);
            System.out.println("Login");
            AccountLogin accountLogin = session.getAccountLogin();

            Uploader uploader = new Uploader(accountLogin);
            
            // OPTIONAL: defaults to not allow overwrite an existing file
            uploader.setOverwriteExistingFiles(true);

            if (useEncryption) {
                useEncryption(uploader);
            }

            // Get fresh upload session
            UploadSession uploadSession = uploader.getUploadSession(localUploadFilePath, remoteFolder);

            // OPTIONAL: If you are resuming an upload which was interrupted by your
            // application crashing, use this alternate UploadSession constructor
            // to load transferDataCommitted() uploadContext from your persistant storage where
            // you last stored it via transferDataCommitted():
            //
            //FileInputStream fis = new FileInputStream("yourSaveState.bin");
            //ObjectInputStream in = new ObjectInputStream(fis);
            //HashMap<String, Object> uploadContext = (HashMap<String, Object>)in.readObject();
            //in.close();
            //fis.close();
            //uploader.setOverwriteExistingFiles(true);
            //UploadSession uploadSession = uploader.getUploadSession(localUploadFilePath, remoteFolder, uploadContext);

            // Setup listener callbacks
            UploadTransferListener listener = new UploadTransferListener(uploadSession);
            
            // OPTIONAL: defaults to 10 as returned by uploader.getUploadPartLengthMB().
            // See setUploadPartLengthMB javadoc warnings if you change the size.
            //uploadSession.setUploadPartLengthMB(10);

            uploadSession.addTransferListener(listener);

            try {
                // Upload the file
                uploadSession.doTransfer();
            } finally {
                uploadSession.removeTransferListener(listener);
            }
        }

        private static void useEncryption(Uploader uploader) throws Exception {
            // To use > 128-bit (16 byte) keys, get JRE extension from
            // http://java.sun.com/javase/downloads/index_jdk5.jsp:
            //   "Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files 5.0"
            // and install per the readme.txt in that zip file.
            byte[] encryptionKey = "087mgTI0M3kPgB1X".getBytes();
            byte[] encryptionIV = "JKlrtMV2BPYWnutX".getBytes();
            uploader.enableEncryption(encryptionKey, encryptionIV);
        }

        // Create and return a standard Java logger which outputs to a file and to console.
        //
        // Applet note from http://download.oracle.com/docs/cd/E17476_01/javase/1.4.2/docs/guide/util/logging/overview.html :
        // Trusted applications are given the appropriate LoggingPermission so they can
        // call any of the logging configuration APIs. Untrusted applets are a different
        // story. Untrusted applets can create and use named Loggers in the normal way,
        // but they are not allowed to change logging control settings, such as adding
        // or removing handlers, or changing log levels. However, untrusted applets are
        // able to create and use their own "anonymous" loggers, using Logger.getAnonymousLogger.
        // These anonymous Loggers are not registered in the global namespace and their
        // methods are not access-checked, allowing even untrusted code to change their
        // logging control settings.
        private static Logger createLogger(String logFilePath) throws IOException {
            Logger logger = Logger.getLogger("NirvanixJavaSDKSample");
            SimpleFormatter formatter = new SimpleFormatter();
            logger.setLevel(Level.WARNING); // set Level.ALL to see all

            // Enable log to text file
            FileHandler fh = new FileHandler(logFilePath, true);
            fh.setLevel(Level.ALL);
            fh.setFormatter(formatter);
            logger.addHandler(fh);

            // Enable log to text console
            ConsoleHandler ch = new ConsoleHandler();
            ch.setLevel(Level.ALL);
            ch.setFormatter(formatter);
            logger.addHandler(ch);

            return logger;
        }
    }

    class UploadTransferListener implements ITransferListener {
        private UploadSession uploadSession;

        UploadTransferListener(UploadSession uploadSession) {
            this.uploadSession = uploadSession;
        }

        public void transferDataCommitted(long sourceOffset, HashMap<String, Object> uploadContext) {
            System.out.format("DataCommitted %d of %d bytes %n%s%n", sourceOffset,
                    this.uploadSession.getSourceLength(), uploadContext.toString());

            // OPTIONAL: If you want the ability to resume an upload which was
            // interrupted by your application exiting or crashing, save uploadContext
            // object here (e.g. serialize it to disk):
            //
            //FileOutputStream fos = new FileOutputStream("yourSaveState.bin");
            //ObjectOutputStream out = new ObjectOutputStream(fos);
            //out.writeObject(uploadContext);
            //out.close();
            //fos.close();

        }

        public void transferProgress(long sourceOffset, long sourceLength) {
            System.out.format("Uploaded %d of %d bytes%n", sourceOffset, sourceLength);
        }

        public void transferRetry(long sourceOffset, long sourceLength, int remainingRetryCount, int msecsToNextAttempt, String reason) {
            System.out.format("Retrying at %d of %d bytes, %d retries remaining, next attempt in %d msecs, reason: %s%n",
                    sourceOffset, sourceLength, remainingRetryCount, msecsToNextAttempt, reason);
        }

        public void transferComplete(long sourceLength, long sinkLength) {
            System.out.format("Upload Completed: source length %d, sink length %d%n", sourceLength, sinkLength);
        }

        public void transferCancelled() {
            System.out.println("Upload Cancelled");
        }
    };

    Please let us know if you have any questions about the new upload object or any questions about the sample.  All functionality is described in this sample.

  • New API Documentation

    The new API documentation contains quite a few new additions and areas that were reworded. Along with these changes a new CodeSamples file has been added that gives simple examples in three different SDKs available on our developer site. This should help make development a bit easier by adding real world examples to enhance learning the API.

    Other changes to the documentation includes a full reorganization of the sections, the goal was to split the documentation into two separate areas. The first area is a overview of the concepts behind our API and overviews behind the structures and protocols needed to access them. The second section is a straight API reference for each API call that includes input, output and common samples for accessing each call. At the bottom of each API reference there is a link to the SDK sample code.

    Please let us know if there are areas you would like to see improved or if there is anything missing in the new API document. Our goal is always to improve your experience and make sure development against our platform is as simple as possible.

    View the API Document.
    View the Code Samples.
  • MD5 Validation for Perl

    Verifying the files you have stored on our servers is an important part of data integrity.  To calculate the MD5 we store on our servers in Perl the quick solution would be to use the built in b64digest output which is part of the Digest::MD5 class.

    For the test file the MD5 being returned by Nirvanix ListFolder is: bNIkeoRicdpTMoPyuN/4rg==


    use Digest::MD5;
    use MIME::Base64;
        my $file = shift || "./somefile.txt";
        open(FILE, $file) or die "Can't open '$file': $!";
        binmode(FILE);
        print Digest::MD5->new->addfile(*FILE)->b64digest, " $file\n";

    output:
    bNIkeoRicdpTMoPyuN/4rg ./somefile.txt

    For some reason this does not produce the expecting padding of: ==

    If you change the script slightly to call the base 64 encoding externally using the digest output (binary MD5) then it will report the correct value.

    use Digest::MD5;
    use MIME::Base64;
        my $file = shift || "./somefile.txt";
        open(FILE, $file) or die "Can't open '$file': $!";
        binmode(FILE);
        print encode_base64(Digest::MD5->new->addfile(*FILE)->digest), " $file\n";

    output:
    bNIkeoRicdpTMoPyuN/4rg==
     ./somefile.txt

    It is important that you do not just add == to the results of the b64digest since this is not always returned (though usually is).  There are cases when the extra bytes are used by the base64 algorithm.

    Posted Jan 13 2010, 07:55 PM by BarryR with 1 comment(s)
    Filed under: , ,
  • SDN URL Redirection

    The dynamic nature of our system allows us to automatically route uploads and downloads to the best node for optimal performance and to ensure system availability during maintenance or heavy load.  This is done by coding URLs to “http://services.nirvanix.com” and allowing the SDN to automatically redirect them to the appropriate node. This is a very important feature that all customers need to follow to ensure the best user experience.
     
    SDN files can be accessed in three ways, specifically by using a: (1) session-based URL, (2) hosted-item URL, or (3) tokenized URL.
     
    1. A session-based URL is codified using the form: “http://services.nirvanix.com/SESSIONTOKEN/PATH/FILENAME”. The SESSIONTOKEN is created by the “Login” API call.  This URL is automatically redirected to the closest active node that contains the requested SDN file.
     
    2. A hosted-file URL is codified using the form: “http://services.nirvanix.com/APPNAME/ACCOUNT/PATH/FILENAME”. Hosted files are created using the “CreateHostedItem” API call. Similar to session-based URLs, this URL is automatically redirected to the closest active node that contains the requested SDN file.

    3. A tokenized URL for an SDN file is generated by the “GetOptimalURLs” API call.  The resulting URL points to a specific node that contains the file when “GetOptimalURLs” was called.  However, if the node is unavailable or the SDN file is subsequently migrated to a different node, then the tokenized URL will return an error and not resolve to the file (even if the URL hasn’t expired yet).  For that reason, we recommend specifying a short expiration period for tokenized URLs and implementing error handling, e.g. regenerating the URL using “GetOptimalURLs” if the file is unavailable.
     
    All three of these methods use IP-based geo-location services to determine the closest active node.  Session-based and hosted-item URLs resolve to the datacenter closest to the client accessing the file.  Tokenized URLs are fixed to the node closest to the IP address passed to the “GetOptimalURLs” API call.

    Uploads are treated similarly to the “GetOptimalURLs” call, such that a token is generated that allows you to upload to a specific node.  You must call “GetStorageNode” or “GetStorageNodeExtended” for each file being uploaded.

  • A Feature Review

    I wanted to go over the different releases over the last year or so to give a quick overview of what has been done.  It can be a challenge to read through our release notes in the forums if you are not already a customer so a roll-up is in order.

     
    The build out of 4 nodes including Japan and Europe.

    CloudNAS for Windows and Linux - We have been focusing on offering significant speed and simplicity to allow you to mount the file system as a network drive.  We have released many new features allowing significant performance improvements and compatibility with many different backup and other server solutions.

    Improved services server - The services servers received a significant improvement to speed and scalability to allow for even faster requests and allowing significant redundancy.

    Output as JSON - We announced output as JSON in our latest release which allows integration with JavaScript making web site integration easier.

    Network Improvements - All of our nodes are continually being improved for network performance.  The overall transfer speed for many of our customers have been many fold improvements for upload and download.  This is something we consider one of the most important features of our service.

    Reliability - While I cannot detail the improvements made in this regard there have been significant changes like improved error handling in the last 6 months.  Features cannot be measured if there is no service running so this is a top concern for us.

    White Lists - The ability to create IP ranges that restrict download of public files to only those IPs.

    Customer Support - The management portal has been enhanced in numerous ways including a ticketing system for support entries and full child account modification and creation.

    This is not a complete list but it is a quick way to catch up on what we have been working on in our drive to add features and provide a great platform to build your application on.

  • Streaming and Seeking in Silverlight with Nirvanix

    Silverlight offers some great video playback features in terms of allowing progressive download and offering a method to index and seek WMV.  This is a great way to allow complete freedom in displaying movie playback while using a progressive download server such as a Nirvanix account.

    The normal buffering that you would see with any progressive downloaded Flash or Silverlight video is still happening, but because the WMV is indexed the player can skip to any point in the video and start buffering / playing from that point.  Once you have your video completed you can upload the file along with some HTML to invoke the player.  The player I have chosen below is quite nice because it is open source, supports the seeking capability and is quite attractive.

    The link below is a demo of movie length content which lets you move the slider to any point.

    Demo:
    http://services.nirvanix.com/NirvanixDrive/barrynirvanix/silverlight/silverlighttest.html

        <object data="data:application/x-silverlight-2," type="application/x-silverlight-2" height="480" width="640">
            <param name="source" value="VideoPlayer.xap">
            <param name="background" value="white">
            <param name="initParams" value="m=http://services.nirvanix.com/App/User/PathtoFile/Video.wmv">
                   <param name="minruntimeversion" value="2.0.31005.0">
            <a href="http://go.microsoft.com/fwlink/?LinkId=124807" style="text-decoration: none;">
                 <img src="http://go.microsoft.com/fwlink/?LinkId=108181" alt="Get Microsoft Silverlight" style="border-style: none;">
            </a>
        </object>

    The object above can be placed in your HTML and will render the video using the VideoPlayer.xap.  You can place the html, player and video in a directory and share that directory to allow the file to be played back.

    The HTML code used to load the player can be downloaded here:

    The SL2 Video Player:
    http://sl2videoplayer.codeplex.com/
  • Video Playback Tutorial

    Often you will want to play the video stored on your Nirvanix account. One of the easiest formats to play back would be FLV since Adobe has done most of the hard work in flash already.  You have the option to either create a player in flash or you can purchase one of the many that are available.  This tutorial will describe how to convert a video in avi format to FLV then upload, transcode and setup a player.

    Uploading the AVI

    For this step we will use the PHP Web Client.  You can find a sample site running at: http://www.nirvanixtest.com/webclient/

    If you intend to use this extensively you should get the source and put the site up on your own server.   To upload we will create a folder in the root called "video"

     

    Transcoding to FLV

    After the upload is complete you can browse to the upload folder and click on the file you want to transcode and click on Transcode Video.

     

    Once you have selected the video to transcode you will need to type the destination file and path along with the transcode quality.

     

    After you click on "Transcode File" It may take a few minutes to process the request in the background.  Depending on the size of the file the time will vary.  You can refresh the destination folder to know when the conversion has completed.  Normally you would have a process in the background listening to the callbackURL which will be called when the video transcode is complete.

     

    Embed Flash in HTML

    Once you have the video converted to FLV you will need to upload a player to this directory.  For this tutorial we have selected the moxiecode FLV player because it is a simple example that follows the logic that most web flash applications use.  You can find a copy at: http://oos.moxiecode.com/flvplayer/ along side: http://blog.deconcept.com/swfobject/ which is used to render the swf object through javascript without too much more work.

    Upload the flvPlayer.swf and swfobject.js to the movies folder along side the FLV.  This will be the movie player you will be using that is embedded in the html page we will be creating.  Once this is done you can upload the html page to the same folder.

     You can view the working html sample here: http://services.nirvanix.com/Movie Viewer/movietest/movies/video.html

  • Silverlight Support

    We are happy to announce we have added the ability to access your content from Silverlight.  The latest release added a file at the root of both servies and the nodes that is a security file allowing Silverlight to connect to and download from both API calls as well as files.  Part of the release was to add the Microsoft suggested Mime-types for hosting Silverlight files as well.  Some restrictions on hosting the Silverlight application may be having to call the node directly to avoid the redirect.

    The security file can be accessed at: http://services.nirvanix.com/clientaccesspolicy.xml and at the root of each node as well.

    I look forward to our Silverlight developers showing off the latest interfaces to access both API calls and file downloads in the future.  Please post a comment if you have a questions or concerns.

     

    Posted Sep 16 2008, 05:44 PM by BarryR with no comments
    Filed under:
  • Callbacks and communicating with processing

     I have received a number of direct questions from Nirvanix users regarding callbacks and thought that I would share the information I sent to them with the entire community. Our processing servers allow you to work on requests that may take a significant amount of time.  This means you can transcode videos that are > 200gb or any number of other supported large file operations.  When processing a large file it's necessary to move these calls to an asynchronous process to avoid keeping an active connection open.  Web interfaces aren't the best mechanism for long running connections and it can take quite a while to transcode a 200gb video depending on the CODEC. This setup requires a way to let you know when we are finished with the operation, this is where callbacks are used.

     When something is added to our queue it will be picked up by our processing servers and when they are done with the request they will make a call to the URL presented in the initial request.  The data returned depends on what operation you were performing.  There will always be a response code passed which if 0 means the command succeeded as expected.  If it is non-zero there will also be an error message. This is necessary because the initial call to the method only adds the request to the queue and does not actually perform the operation. You should receive two separate callbacks, the first is a notification that the file has been added to the queue successfully (or un successfully) and the second is notification that the file has been processed successfully (or unsuccessfully).  Cases where the initial notification would not be 0 include paths already existing, users reaching their set limits or other problems.

    The callback URL you submit to us is in the following format http://www.mysite.com/mycallback.php which would reside on your site.

    Once a file has completed processing you will receive the following information:

    NVX.returnCode an integer representing the status of the upload. Did it fail? Succedd? For valid response codes: see standard return codes; return code 0 means no error.
    NVX.errorMessage a string describing the error message associated with any error being returned.
    NVX.absolutePath a string describing the absolute path of the file that was uploaded.
    NVX.sizeBytes a long describing the size in bytes of the file that was uploaded.
     

    Transcoding Audio / Video

    The transcode calls can take some time to convert the files depending on the input and output formats as well as the quality settings.  For this callback the operation will only return if the call was successful as well as any errors that have occurred during processing.  If you are converting to FLV / MP3 formats you can use this callback to determine when you should make the files visible to the users of your system.

    Image Resize / Rotate

    The image calls usually occur very quickly and will return if the call was successful as well as any errors that have occurred during processing.  Possible reasons for failure would be that the submitted format is unknown to us.  Often this call is used to swap a preview image with a generated thumbnail.  This can be a good queue for your system to present a resized image instead of a stock icon if you have a way to show previews of image files on your system.

    Sideload

    For sideloads we are actually going out on the web and making requests of external sites which can have significant overhead if the site we are contacting should be slow or unresponsive.  It is critical that you capture sideload errors since this is calling to an external system which can often behave in unexpected ways.  The sideload call is a great way to save external data into your account without having to proxy the files through your own system.

    Upload

    Uploads are different in what they return so you can do post processing on a file after it has been uploaded.  Often systems using our web services allow users to upload directly to our servers which allows you to avoid proxying files and letting your customers upload the files directly to us.  The challenge is having a way to update your system or local database with the files those users have uploaded.  The callback can be used as a convenient way to keep your local system informed of when a user uploads their own file.  Another use of this callback is to execute a secondary action when the upload is complete.  You can check the file for the extension to see if its allowed, generate a thumbnail on an image or any number of operations.

     Sample Callback Handler in PHP

     

    <?php 

    $myFile = "requestlog.txt";
    $fh = fopen($myFile, 'a') or die("can't open file");
    fwrite($fh, 'Requesting URI: ' . $_SERVER['REQUEST_URI'] . "\r\n");
    fwrite($fh, 'Request Time: ' . date("D M j G:i:s T Y") . "\r\n");
    fwrite($fh, 'Query String: ' . $_SERVER['QUERY_STRING'] . "\r\n\r\n");

    fclose($fh);

    ?>
    The handler will append the results of each callback to a file called requestlog.txt.  Make sure your web server (PHP) has write privileges to that file.
More Posts Next page »