I need to transfer a file and I don't know the total size in advance.

I create a transfer with  the following code:

public Transfer GetTransfer(long size, ref string transfertoken)

{ 

  UploadNode uploadNode = Utilities.GetUploadNode(this.accountLogin, size);
        Transfer transfer = new Transfer();

        UriBuilder uriBuilder = new UriBuilder(transfer.Url);
        uriBuilder.Host = uploadNode.Host;
        transfer.Url = uriBuilder.ToString();
        transfertoken = uploadNode.AccessToken;

        return transfer;

}
 

 I then call a function like this to write data to the remote server.

buffer is a chunk of  up to 64K bytes, EOF is set true when I am ready to stop sending. 

public void dowork(bool EOF,byte[] buffer, string path) 

            m_transfer.AppendFile(m_transfertoken, path , buffer, EOF);

}

 
Everything works just fine until EOF is set to true, at which point the program HANGS and I must reboot my computer to get it back.

 I must be missing something.

HELP!!!
 

There is a C# snippet for doing uploads that was released on 7-20 which does all of this work for you.  Also I would suggest using the partial file upload which uses http as the transport which will save you some bandwidth in the transfer since soap has to encode the content and the overhead of the soap message itself.  This library does all of that for you.

http://developer.nirvanix.com/files/folders/csharpsnippets/entry580.aspx

Also we have wrapped the upload functionality found in this snippet as well as the rest of the API calls through rest in a very efficient wrapper in the form of the C# Flat SDK.  You can find it here:

http://developer.nirvanix.com/files/folders/reference_code__applications/entry692.aspx

Regards,
   Barry R.

Thanks. Was using that snippet and ran into the roadblock espoused in my first sentence. I don't know the total length of the file I am going to write when I start writing. If you choose an arbitrarily large number, and don't send that exact amount, you get an exception.

 I also have the flat sdk and most of the other downloads available. 

All of the methods in the httpupload.cs presuppose that you know the total length of the data you will be writing, in advance.

If you massage the headers and increase the file size on each write, IE 0-99/100, then 100-199/200 you get a response header that says you uploaded (X) files,where x is the number of writes you made to the stream. You also get a response code of 0 (success) and NO FILE APPEARS on the server.

 There must be a simple way to open a stream on the Nirvanix server and write to it until I am done, without knowing the file size in advance (think pipe). If not, there is a glaring hole in the API. :)

 I don't need random access, updates etc. I just need to be able to easily stream data onto a remote file from a pipe or other non-fixed size input.

Can this be done? Pardon my tone, It's nearly 6:00 am and I've been at this for 14 hours.

JP

 

 Using the upload the total size header only needs to be larger than the bytes sent.  for your example:

0-99/101 would work since you are saying you expect 1 more byte so it will keep the upload open for more data.  The next would be 100-199/200 if you wanted to close the file or 100-199/201 to write more.

Regards,
    Barry R.

 Well, that almost works. One small problem, since I don't know the total size of the file to be written, I can't properly set the

httpWebRequest.ContentLength

property.

Some interesting notes:

1. Leave it alone. The file does not actually start transmittng until you call requestStream.Close(). This leads to huge memory consumption and usually an aborted  upload (also the request times out a lot)
   
2. 2. Set it to a large arbitrary number, say, 2GB., Exception on every write. (Out of memory)
   
3.  I am not a web protocol expert. I'm much more of a hardware/driver type guy. New to C# as well. OUCH.

 Currently i am just WAGGING it at 512M for testing.

UPDATE: That does not work. I set requestStream.SendChunked to true and then data does begin to flow as it is written into the requestStream

But, always, after around 900K of data has been sent, I get the following exception:

Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host. 

 MORE Data:

Leaving the ContentLength property at default results in :

<Response>
    <ResponseCode>70121</ResponseCode>
    <ErrorMessage>Invalid upload request: total bytes uploaded does not match content-length header.</ErrorMessage>
</Response>

So, it looks like there is no easy way to write or append to a file UNLESS YOU KNOW EXACTLY how much you will be writing. PERIOD. 

the Update method requires a callback URL, which won't work for us. The soap.AppendFile command hangs like a big dog and the HTTP upload method won't work either. 

This means that I must either cache the entire stream in memory before writing (kinda defeats the purpose), or choose another bulk data provider and tell the project manager we are up the proverbial creek, chunk a months coding efforts and start over.

This just doesn't strike me as being that hard. Why is it?

The AppendFile method doesn't hang in my tests, also the FTP Proxy is using the AppendFile method for its uploads and its used by quite a few customers without issue.

For the  upload sample you can define your own file part and pass it in.  For http uploads you should build the buffer you are going to send then pass it.  You can call the HTTP Upload method multiple times, essentially this means you should load about 1mb (a good size for most operations) into a buffer, call the upload then close the connection.  Repeat this in a loop until you are ready to close the file. 

Perhaps you can post your full code sample so it can be debugged since this is not a server capability issue, api ability issue and doesn't seem to be a problem with AppendFile.

Best,
   Barry R.

 Here is a sample that uses the HTTP Upload snippet to upload two different chunks of data.  The first one creates a new file and does a post leaving the file open for more updates.  The second call finishes the file and closes it.

.. SNIP ..

            // Get the upload node
            UploadNode node = imfs.GetUploadNode(sessionToken, new FileInfo(TEXT_FILE).Length);
            // Format the URL with
            string url = "http://" + node.IPAddress + "/Upload.ashx?uploadToken=" + node.AccessToken + "&destFolderPath=/httpupload/";

            List<FilePart> files = new List<FilePart>();

            FilePart filePart = new FilePart();
            filePart.binaryStream = new FileStream(TEXT_FILE, FileMode.Open, FileAccess.Read);
            filePart.filename = TEXT_FILE;
            long fileLen = new FileInfo(TEXT_FILE).Length;
            filePart.startPos = 0;
            // This gets the file size and divides by 2 leaving some data for later.
            filePart.endPos = fileLen / 2 - 1;
            filePart.totalFileLength = fileLen;
            files.Add(filePart);

            HTTPUpload.MultiPartUpload(url, files, null);

            files.Clear();
            filePart.startPos = fileLen / 2;
            // Finish off the file starting at the last uploaded position and at the end of the file closing it.
            filePart.endPos = fileLen - 1;
            files.Add(filePart);

            HTTPUpload.MultiPartUpload(url, files, null);

I have verified this code works and I have used similar code to buffer logs into a memory stream 1 mb at a time then upload them moving on to the next chunk that is buffered until the memory buffer is ready to upload the next part.  This seems to be what you are trying to accomplish.  let me know if you need a different sort of example.

Regards,
   Barry R.

Barry sez:

 Here is a sample that uses the HTTP Upload snippet to upload two different chunks of data.  The first one creates a new file and does a post leaving the file open for more updates.  The second call finishes the file and closes it.

 Yup, Note that the first filepart (and the second) contain the total amount of data to be uploaded. The size of the data to be uploaded is known in advance. Am I missing something? The filepart.totalLength is used to help calculate the httpwebRequest.ContentLength.

Barry sez:

I have verified this code works and I have used similar code to buffer logs into a memory stream 1 mb at a time then upload them moving on to the next chunk that is buffered until the memory buffer is ready to upload the next part.  This seems to be what you are trying to accomplish.  let me know if you need a different sort of example.

 I would love to see your buffering code please.

Thanks,

JP 

Well, it finally hit me that even though the MultipartUpload code used a ContentLength parameter, it was only for that chunk of file.

If you set the filepart.TotalLength to one more than the filepart length, except for the last chunk to be uploaded, it will work.

 It's working now., Except for the server randomly dropping chunks of file. :)

IE, occasionally I get an error message that my chunks are out of order, usually after a long pause during upload of the previous chunk.

 Thanx,

JP
 

 I suspect the order problem could be due to multi-threading and some of the threads are hopping in front of the other.  Let me know if this sounds like a possibility.

Regards,
    Barry R.

 While the application is multithreaded, a single thread handles all uploads/downloads currently.

What's bad, is I get a positive ack from the server that the missing block was received.

I have some ideas to try. Will let you know.

JP