Dear Nirvanix Team:

One huge feature that I think would help you to set yourself apart from Amazon and other up-and-coming storage providers is the ability for our client-side software applications to do delta level backups and process them with server-side assistance.  Please allow me to explain:

Many of us are using the service to do backups of our computers or to store archived data or even to store huge graphics files, video, and other items that can take a LOOONG time to upload.  So, what I would find HUGELY attractive to your service is functionality that is similar to rsync where it uses both the server-side and client to pass delta information efficiently.

If you could provide us with the ability to upload a huge file (or even small files) and we could easily pass back block-level deltas and upload only the portions that the server-side needs and then you patch them.. well I think I would come down there and give each of you a great big KISS!

I'm going to guess that many others would use this feature and because Amazon and others don't offer it.. well.. you'll be WAAAAAY ahead of the game and offering all of us a service that we can all use!  Besides, who wants to pay for all that bandwidth when we don't really need to?

You guys are doing a great job and I think this type of feature will be a win-win for everyone.  Keep up the excellent work!

Sincerely,

Joshua

 We have this functionality already available.  Look in the documentation for "Partial File Update" for a full description of how to use it along with some examples.  The basic implementation lets you treat the upload the same way you would as partial file upload via http but instead upload just a portion of the new file that has changed.  The original idea was spawned for exactly the reason you describe.  We wanted to make sure differential backups could be performed on a file without sending the whole file.  The difficult part of course is performing the diff and knowing what has changed without keeping two copies at the client but I think that is for those who are in the backup space to know far better than we will.

Please let me know if you have any questions on how this is implemented or if you have any other questions.  You can find a C# example of the Partial File Update which shows how to change a file here:

http://developer.nirvanix.com/files/folders/csharpsnippets/default.aspx

Regards,
    Barry R.
 

Barry:

I was aware of the upload partial file but as a noob c# programmer it was determine what portion to upload!?  I mean.. I can see how your software could combine the data file to "update it" with the latest bits but how do we know what bits to send?  I think we need to see this all work from start to finish with a code example from you guys as it's not making sense to me.  Is this something y'all can do for us!?

I think others would concur.  Thanks!

- Joshua

 This, I feel, is one of the toughest problems facing backup companies.  How to quickly, reliably get a difference between files and know only what needs to be updated without duplication of data and letting it happen quickly.  Unfortunately, due to the raw difficulty mixed with our lack of expertise in this area, a sample probably wouldn't help much.  We can certainly determine if a whole file is different but trying to determine if portions of a file is truly a difficult task.

Just to get you started you could look at some binary difference code combined with the update code.  I'm afraid we don't have any more of an idea than anyone else of what the best way to do this is and it would be a work of innovation and not just something we can create an example for.

Take a look at this for a binary diff: http://www.codeproject.com/KB/recipes/diffengine.aspx 

Best Regards,
     Barry R.

I understand the difficulty Barry.  I appreciate you taking the time!

How about something simple like supporting an rsync process on the back-end, sort of like rsync.net ?

We really need something in this area too. Getting stuff up to Nirvanix is the biggest challenge. 

--Cameron

If you are using our CloudNAS product in cached mode you can use RSync.  We test this with every build of CloudNAS since there are many users who use it in this way.

Regards,
      Barry R. 

Our ultimate requirement is to be able to replicate/sync portions of the SDN filesystem to a desktop. Essentially, to be able to cache a Nirvanix folder on the desktop with Bi-directional sync. We are trying to figure out how we are going to piece this together. Maybe you have some ideas?

Question: Is the CloudNAS code something that would be suitable for running on individual client machines?

Setting up our own server and using CloudNAS is certainly one way to go, and will probably be, by default, what we have to do. However, the things that concern me about using CloudNAS this way are:

1) I'm going to be paying for double bandwidth. By comparison, Amazon WS does not charge for bandwidth between its EC2 and S3 nodes.

2) I'm concerned about the 4 minute delay in seeing new/changed files via CloudNAS. Its hard to anticipate the issues this will create, but it will certainly cause some users confusion.

3) I'm still a little unclear on how many applications or child-accounts can reasonably be mounted on a *nix box, and how we will manage this.

Thanks,
Cameron

 What I am not able to piece together from our correspondence is the type of system you are building.  It seems you are trying to put together a realtime distributed file system yet trying to do this with backup tools.  RSync even on the best of systems is never a realtime process and usually takes some time to replicate changes to all places.  You bring up the 4 minute mark as a point of contention but even using a traditional RSync file server over http is going to have at least that much latency.  If I were to guess it looks like you are trying to build a instant distribution to many nodes, something RSync may not going to be good at either since its traditional role has been to replicate ftp sites across universities and other similar activities. 

I just want to make sure you have a realistic understanding of how we designed our system.  Our primary goal for our system is the ability to store large amounts of data over a long period of time.  This is ideal for archival purposes.  We also focus on light delivery as a means to distribute to your end users but have not built out a significant edge network that a tradtitional CDN would have.  This limits some possibilities for doing real-time actions to a large number of users.  The other primary use for our system is moving files across the globe to many locations so if you need distribution you can use that for availability and to lower the latency for accessing your files.

One thing we are not focused on is realtime byte for byte distribution to many end users.  This is a role that traditional CDNs do quite well through a large number of edge servers and one source server for distribution.  I'm not sure of your use case but it seems like there are two parts, one is to perform regular backups, the other is to mass distribute files from what I have read.  Can you provide for me a better explanation so we can focus on how to accomplish what you need?

Best,
     Barry R.

Hi Barry,

Absolutely, although I think I'll take it offline as this is a public forum. I will email you and Cory. Anyone interested in the outcome can email me at cameron@shotrunner.com.

Cheers,
Cameron