I have a script based on the Perl REST and SOAP packages found here and I'm trying to upload a 7MB file in chunks.  I've varied the chunk size from 1 MB to 10KB and I get the same results each time.

 
The script does the following:

• -Logs in retreiving the session token
• -Deletes the file being uploaded 
• -Calls SOAP::IMFS::GetUploadNode($masterSessionToken,$fileSize,0);  and gets an upload token.
• -Calls SOAP::Transfer::AppendFile($uploadToken,$fn,$dpath,$chunkSize,0);

While in AppendFile, the first chunk is uploaded ok it seems, however the second chunk results in a fault "The path already exists." (70004)  and an HTTP 500 status. 

 
I'm basically using this code as it was downloaded and don't understand why it's not working.  The logic seems fine to me.  I really would like to get this method to work instead of the full file upload in one call.  Is there a problem with this supplied code?  Perhaps there is a better way to do this or my methodology, though simple, is wrong?

 
Thanks for any help.  What else do you need to know?


 

rbranstetter:

While in AppendFile, the first chunk is uploaded ok it seems, however the second chunk results in a fault "The path already exists." (70004)  and an HTTP 500 status.

This seems like you are not using the same upload token.  The uploadToken is a contract for the life of one upload of one file.  This means you should generate one uploadToken for all appendFile calls for a single file.

Let me know if this is not the case.

Regards,
   Barry R.

I've verified that the uploadToken value is consistent between calls to the Transfer object.  The error is occurring during the second pass of the for loop below.

sub AppendFile {

    #Takes Upload Token, Source File Name, Destination File Name, No Of bytes to Upload on each iteration and the expected response code as parameters.

    ########################## Method Invocation Parameters###########################
   
    my ($uploadToken,$sourceFile,$destFile,$expected,%options,$AppendFile,$trueFlag,$falseFlag);
    my ($res,$req,$bytesRead);
       
    ########################## File Variables ########################################
    my($fileSize, $noOfBytes,$content,$fileData,$endOfFile);

   
    ($uploadToken,$sourceFile,$destFile,$noOfBytes,$expected)=@_;


    %options = (proxy=>'http://node1.nirvanix.com/ws/transfer.asmx',
                uri=>'http://transfer.nirvanix.com/ws/Transfer',
                on_action=>sub {join '/','http://transfer.nirvanix.com/ws/Transfer',$_[1]});

   
    $AppendFile = SOAP::Data->name('AppendFile')->attr({xmlns=>'http://transfer.nirvanix.com/ws/Transfer'});
    $destFile = SOAP::Data->name(path=>$destFile)->uri('http://transfer.nirvanix.com/ws/Transfer');
    $trueFlag  = SOAP::Data->name(endOfFile=>'true')->uri('http://transfer.nirvanix.com/ws/Transfer');
    $falseFlag = SOAP::Data->name(endOfFile=>'false')->uri('http://transfer.nirvanix.com/ws/Transfer');
    $uploadToken= SOAP::Data->name(uploadToken=>$uploadToken)->uri('http://transfer.nirvanix.com/ws/Transfer');
    $req = SOAP::Lite->new(%options);
    if (-e $sourceFile) {
           
         print "File Exists \n";
         my $fileSize = stat("$sourceFile")->size;
         print "Size: $fileSize\n";
         if ($fileSize>0) {
            open (INFILE,'<',$sourceFile)||die "Unable to open file\n";
            binmode(INFILE);
           
            for (my $j=$noOfBytes;$j<$fileSize;$j+=$noOfBytes) {
                    print "In For\n";
                 $bytesRead=read(INFILE,$fileData,$noOfBytes);
                 $fileData=SOAP::Data->name(fileData=>$fileData)->type('base64')->uri('http://transfer.nirvanix.com/ws/Transfer');
                 $res = $req->call($AppendFile,$uploadToken,$destFile,$fileData,$falseFlag);

                    if ($res->fault) {

                        print " \t################################# In For Loop ###############################\n";   
                        print $res->faultstring,"\n";
                        last;
                    }
            }

            if (!$res->fault) {
                $bytesRead=read(INFILE,$fileData,$noOfBytes);
                $fileData=SOAP::Data->name(fileData=>$fileData)->type('base64')->uri('http://transfer.nirvanix.com/ws/Transfer');
                $res = $req->call($AppendFile,$uploadToken,$destFile,$fileData,$trueFlag);
            }

            if ($res->fault) {
                print $res->faultstring,"\n";
            }
            close INFILE;
         }
         else {
             print ("No Data In File");
         }

    }
    else {
        print " File Doesn't  Exist\n";
    }

}
 

 I don't see anything particularly wrong with your logic in this code.  There was one potential error that our team found:

open (INFILE,'<',$sourceFile)||die "Unable to open file\n";
            binmode(INFILE);
           
            for (my $j=$noOfBytes;$j<$fileSize;$j+=$noOfBytes) {
                    print "In For\n";
                 $bytesRead=read(INFILE,$fileData,$noOfBytes);

Did you mean to set your loop with $noOfBytes or should it be set to 0?

for (my $j=0;$j<$fileSize;$j+=$noOfBytes) {

Regards,
    Barry R.

That does seem a bit odd.  As this function is very close to the reference Perl in the download section, I didn't really think it was a problem.  I've tried it with that change made and the result is still the same though.

 It really is a simple looping mechanism and according to my understanding as long as the call to AppendFile is done with the False flag, it should be appending each chunk to the previous one.  The only time the True flag is used is outside of the loop, which is where the uploadToken should be closed/invalid.

I'm really confused and need this to function.  I've tried to use the Upload call as well and while it does work, it's extremely inefficient with large files and unuseable in our application.

 I will try to get a working example of a perl large file uploader as soon as possible since it is not directly possible for us to debug external code.  I don't see anything directly wrong with your code but would rather be calling partial file upload.  When the perl scripts were originally written only SOAP had the ability to upload large files.  Since we have the ability to do this over a standard post I would like to get a example of that instead since it will be much more efficient.

I am moving this thread to perl forum since its related to perl coding and not the general service.
 

I've just found the nput Upload script for Bash and it looks like that can satisfy our need temporarily.  I'd prefer to use the AppendFile method still so I will anticipate your post.