Linux FTP Proxy

Last post 03-31-2008 6:22 PM by BarryR. 15 replies.
Page 1 of 2 (16 items) 1 2 Next >
Sort Posts: Previous Next
  • 03-20-2008 5:30 PM

    Linux FTP Proxy

    Is there a ftp proxy. I have a fedora core 4 server. Is there any tool that i could use to setup a ftp proxy on my linux server.

     There are no other tools for linux file transfer.


     

    Filed under:
  • 03-20-2008 6:36 PM In reply to

    • BarryR
    • Top 10 Contributor
    • Joined on 07-20-2007
    • San Diego
    • Posts 576

    Re: Linux FTP Proxy

    I haven't been successful porting the FTP Proxy yet but the 2.0 version of the FTP Proxy should work with Mono and be portable.  I can take a look tonight to give it a second try.

    Regards,
         Barry R.

    IM Support (Feel free to add me)

    MSN: barryruffner@msn.com
    Gmail: barryruffner@gmail.com
  • 03-21-2008 2:13 PM In reply to

    • BarryR
    • Top 10 Contributor
    • Joined on 07-20-2007
    • San Diego
    • Posts 576

    Re: Linux FTP Proxy

     While I wasn't able to port the 1.0 version of the FTP Proxy I did work on a simple bash script using cURL that will allow mass upload of your files.  For this to work you will need to generate your own Upload token. You will need bash and cURL installed on the Linux machine for this to work.

    Save the script below to your Linux machine as "nput"

    #!/bin/bash
    #iterates through a directory, uploading each file to your Nirvanix account.

    MIN_ARGS=3
    E_OPTERROR=65
    if [ $# -lt $MIN_ARGS ]
    then
    echo "Usage: `basename $0` path uploadDir uploadToken"
    exit $E_OPTERROR
    fi
    # use find to go through every directory and upload the files to node 1
    in a single directory.  This will flatten out the dir structure!

    find $1 -type f -exec curl -F 'upload=@{}' 'http://node1.nirvanix.com/Upload.ashx?uploadToken='$3'&destFolderPath='$2 \;

     To do this you can call the Login REST call at:


    http://services.nirvanix.com/ws/authentication/login.ashx?appKey=YOURAPPKEY&username=YOURUSER&password=YOURPASS

    You can then get the session token from this and call GetStorageNode:

     http://services.nirvanix.com/ws/IMFS/getStorageNodeExtended.ashx?sessionToken=YOURSESSIONTOKEN&sizeBytes=100000&ipRestricted=false&destFolderPath=/YOURUPLOADDIR/

    This will return an UploadToken.  To call the script with the results from this do:

    ./nput ./pathtoupload /YOURUPLOADDIR/ UPLOADTOKEN

    Let me know how this works.

    Regards,
        Barry R.
    IM Support (Feel free to add me)

    MSN: barryruffner@msn.com
    Gmail: barryruffner@gmail.com
    Filed under: , , ,
  • 03-21-2008 2:54 PM In reply to

    Re: Linux FTP Proxy

    Thanks for the script. 

     I tried to run your script and the following is the outcome.

     

    ./nput.sh /mnt/ss.avi trans xD286hlb~PDBs_xhkE4~dtzkTgiB~XXXXXXXXX~dYd4Z-xsR7efNSRFsQ             
    /root/./nput.sh: line 3: account.: command not found
    Usage: nput.sh path uploadDir uploadToken
    /root/./nput.sh: line 5: exit: fi: numeric argument required
    [root@domU-12-31-38-00-3E-28 mnt]#

     

     

    Filed under:
  • 03-21-2008 4:21 PM In reply to

    • BarryR
    • Top 10 Contributor
    • Joined on 07-20-2007
    • San Diego
    • Posts 576

    Re: Linux FTP Proxy

     I'm not sure what the problem would be since this is just a simple bash script.  I have tested this on Unbuntu, Cygwin (under windows), and debian with no problem.  If I were going to guess it looks like its executing it in a shell other than bash since line 3 is just setting a variable.  You could try running bash to see if its working properly by executing /bin/bash

    Let me know if you find anything else since I'm not sure how much additional help I can be.

    Regards,
        Barry R.

    IM Support (Feel free to add me)

    MSN: barryruffner@msn.com
    Gmail: barryruffner@gmail.com
  • 03-21-2008 9:36 PM In reply to

    Re: Linux FTP Proxy

    Sorry my mistake. I had the wrong parameteres. It works great. Just transfered a file to nirvanix. And the script is great. Would start to test it out with a bigger folder.



    Thanks for the help

    Regards
    Hareem.
     

    Filed under:
  • 03-21-2008 9:51 PM In reply to

    Re: Linux FTP Proxy

     There are some problems with the file upload. The token gets expired. I was trying to upload a folder that contained 1.1GB of data and i get the following errors. The script works fine.

    <?xml version="1.0" encoding="utf-8"?>
    <Response>
            <ResponseCode>0</ResponseCode>
            <FilesUploaded>1</FilesUploaded>
            <BytesUploaded>122400014</BytesUploaded>
    </Response>curl: (26) failed creating formpost data
    curl: (26) failed creating formpost data
    <?xml version="1.0" encoding="utf-8"?>
    <Response>
            <ResponseCode>0</ResponseCode>
            <FilesUploaded>1</FilesUploaded>
            <BytesUploaded>83350826</BytesUploaded>
    </Response><?xml version="1.0" encoding="utf-8"?>
    <Response>
            <ResponseCode>80102</ResponseCode>
            <ErrorMessage>Expired Access Token</ErrorMessage>

     

    I would really appreciate it if you could help me out by some how increasing access token expiration

            
     

    Filed under:
  • 03-21-2008 10:10 PM In reply to

    • BarryR
    • Top 10 Contributor
    • Joined on 07-20-2007
    • San Diego
    • Posts 576

    Re: Linux FTP Proxy

    The getStorageNodeExtended call has this functionality built in.  You can add the parameters firstByteExpiration and lastByteExpiration to the call

     http://services.nirvanix.com/ws/IMFS/getStorageNodeExtended.ashx?sessionToken=YOURSESSIONTOKEN&

    sizeBytes=100000&ipRestricted=false&destFolderPath=/YOURUPLOADDIR/&firstByteExpiration=86400&lastByteExpiration=86400

    The above will allow the token to last 24 hours. You can read more about the call at the link below.

    http://developer.nirvanix.com/sitefiles/1000/API.html#_TocGetStorageNodeExtended

     Regards,
        Barry R.

    IM Support (Feel free to add me)

    MSN: barryruffner@msn.com
    Gmail: barryruffner@gmail.com
  • 03-22-2008 9:24 PM In reply to

    • BarryR
    • Top 10 Contributor
    • Joined on 07-20-2007
    • San Diego
    • Posts 576

    Re: Linux FTP Proxy

     I have improved the script a little by allowing it to also store the directory where it originates from so you can upload entire trees.  This means it will replicate the local linux file structure remotely.  I will work on a final version that will be released as a full app that does the token lookup bit this is quite a bit more work so it may take a while.

    #!/bin/bash
    #iterates through a directory, uploading each file to your Nirvanix account.

    MIN_ARGS=3
    E_OPTERROR=65

    if [ $# -lt $MIN_ARGS ]
    then
        echo "Usage: `basename $0` path uploadDir uploadToken"
            exit $E_OPTERROR
    fi

    # use find to go through every directory and upload the files to node 1 in tree at Nirvanix.
    for i in `find $1 -type f`;
    do
        upload_dir=$(dirname $i);
        curl -F 'upload=@'${i} 'http://node1.nirvanix.com/Upload.ashx?uploadToken='$3'&destFolderPath='$upload_dir;
    done

    Let  me know if you have any comments on this one.

    Regards,
         Barry R.

    IM Support (Feel free to add me)

    MSN: barryruffner@msn.com
    Gmail: barryruffner@gmail.com
    Filed under: ,
  • 03-24-2008 5:30 AM In reply to

    Re: Linux FTP Proxy

     The following errors are coming up. They did not occur before.

     I entered the following command:

    /root/./nput /mnt/dhoom3 trans/ dPR4Aou7~W8ZEYJzIxd~oByrKalx~YJaS0FHAm-~JODsy9GYTmIsO6oWqw

    Curl Result.

    <?xml version="1.0" encoding="utf-8"?>
    <Response>
            <ResponseCode>80023</ResponseCode>
            <ErrorMessage>Invalid Access Token: dPR4Aou7~W8ZEYJzIxd~oByrKalx~YJaS0FHAm-~JODsy9GYTmIsO6oWqw</ErrorMessage>
    </Response>curl: (26) failed creating formpost data
    curl: (26) failed creating formpost data

    Filed under:
  • 03-24-2008 3:14 PM In reply to

    • BarryR
    • Top 10 Contributor
    • Joined on 07-20-2007
    • San Diego
    • Posts 576

    Re: Linux FTP Proxy

    It looks like there was a problem with how I setup the URL Above for getstoragenodeextended with the destination folder path.  I will look more into it in a bit but here is a corrected URL that will work:

     http://services.nirvanix.com/ws/IMFS/getStorageNodeExtended.ashx?sessionToken=XXXXXXXXX&sizeBytes=100000&ipRestricted=true&destFolderPath=/&fileOverwrite=true&firstByteExpiration=60000&lastByteExpiration=259200

     

    Also for the call to the script you should pass the directory without the ./ in front or it will create a ./ dir.  The example below im passing desktop and that will be the root dir you upload to.

    ./nput Desktop/ /updir/ XYZTOKEN

    Regards,
         Barry R.

    IM Support (Feel free to add me)

    MSN: barryruffner@msn.com
    Gmail: barryruffner@gmail.com
  • 03-24-2008 3:31 PM In reply to

    • BarryR
    • Top 10 Contributor
    • Joined on 07-20-2007
    • San Diego
    • Posts 576

    Re: Linux FTP Proxy

     I found the problem, I wasn't pre-pending the upload directory so when it went to use the token the upload directory mismatched and failed validation.  The script below fixes the problem.

    #!/bin/bash
    #iterates through a directory, uploading each file to your Nirvanix account.

    MIN_ARGS=3
    E_OPTERROR=65

    if [ $# -lt $MIN_ARGS ]
    then
        echo "Usage: `basename $0` path uploadDir uploadToken"
            exit $E_OPTERROR
    fi

    # use find to go through every directory and upload the files to node 1 to a tree.
    for i in `find $1 -type f`;
    do
        upload_dir=$(dirname $i);
        curl -F 'upload=@'${i} 'http://node1.nirvanix.com/Upload.ashx?uploadToken='$3'&destFolderPath='$2$upload_dir;
    done

    This should now put the entire tree under the directory you specify.

    Regards,
         Barry R.

    IM Support (Feel free to add me)

    MSN: barryruffner@msn.com
    Gmail: barryruffner@gmail.com
  • 03-24-2008 5:14 PM In reply to

    Re: Linux FTP Proxy

    Its still the same problem. I followed your instructions and the following errors still occur:

     

    <?xml version="1.0" encoding="utf-8"?>
    <Response>
            <ResponseCode>80023</ResponseCode>
            <ErrorMessage>Invalid Access Token: dFYFes8g~RYoJvDc3wW~XXXXXX~Uk0KxvVBIR~jxZh-zYoWQFIl9Ugdw</ErrorMessage>
    </Response>curl: (26) failed creating formpost data
    curl: (26) failed creating formpost data

    [3]+  Stopped                 /root/./nput /mnt/dhoom3 trans/ dFYFes8g~RYoJvDc3wW~XXXXXX~Uk0KxvVBIR~jxZh-zYoWQFIl9Ugdw

     

    I have no idea why this is happening.

    Filed under:
  • 03-24-2008 6:14 PM In reply to

    • BarryR
    • Top 10 Contributor
    • Joined on 07-20-2007
    • San Diego
    • Posts 576

    Re: Linux FTP Proxy

     I think because it is uploading to each directory individually the directory name is colliding.  The only way around this until I modify the code to automatically generate the upload token is to use the regular GetStorageNode with the time limits.  I will see what I can do to get the full version created.

    Regards,
        Barry R.
     

    IM Support (Feel free to add me)

    MSN: barryruffner@msn.com
    Gmail: barryruffner@gmail.com
  • 03-30-2008 11:38 PM In reply to

    Re: Linux FTP Proxy

     hmmm version 2.0 sounds interesting... any ETA?

    Filed under:
Page 1 of 2 (16 items) 1 2 Next >