HPC Beta - Logo
Dashdark X Webflow Template - Icon
Overview
Getting Started
Core Concepts
Data Storage
Cluster Control
Snapshot
Guide
Convergence plot
File tailing
Use SSH
Whitelist / blacklist
Launch a simulation
Tutorials
OpenFOAM Motorbike
StarCCM+ on Qarnot HPC
Data and Licenses
Configure IP Whitelisting and Port Forwarding to allow Qarnot to access to your self-hosted licenses
Configuring your licence server to use it with Qarnot
Manage your data using rclone cli
Manage your data with a dedicated UI
Essayez dès maintenant !
HPC Beta - Logo

Manage your data using rclone

rclone is a command-line program for managing files on cloud storage. This guide covers installation methods for Windows, macOS, and Linux, including options for users without administrative privileges.

Requirements

  • Basic knowledge of the terminal.
    • To copy or paste information into your terminal, use your mouse or Ctrl+Shift+C/V.
  • Using a compatible operating system (Windows, macOS or Linux)

Install rclone

rclone is a portable executable, meaning you don’t necessarily need admin rights to run it.

  1. Download: Download the rclone version corresponding to your operating system (Linux, macOS, Windows) from https://rclone.org/downloads/.
  2. Installation: Follow the official installation instructions for your operating system.
  3. Linux/macOS without Admin Rights (Optional):
    • Unzip: unzip rclone-*.zip
    • Create bin Directory: mkdir -p ~/bin
    • Copy rclone: cp rclone-*/rclone ~/bin/
    • Permissions: chmod 755 ~/bin/rclone
    • Add to PATH: Add the following line to your ~/.bash_profile or ~/.zshrc file (depending on your shell): export PATH=$PATH:~/bin
    • Reload Shell: source ~/.bash_profile (or source ~/.zshrc)

For windows users you can follow the step by step guide here after

Use rclone (CLI) to Connect to Qarnot

  1. Run config: Type rclone configto enter the interactive configuration session.
  2. Create New Config:
    • n/s/q> n (New Config)
    • name> Enter a configuration name (e.g., "qarnot-storage").
    • Storage> s3 (S3 protocol).
    • provider> Ceph (Ceph storage implementation).
    • env_auth> false (to avoid using AWS credentials from environment variables).
    • access_key_id> Your Qarnot account email address.
    • secret_access_key> Your Qarnot API token (available at https://tasq.qarnot.com/settings/access-token).
    • region> (Leave empty).
    • endpoint> https://storage.qarnot.com
    • location_constraint> (Leave empty).
    • acl> (Leave empty).
    • server_side_encryption> (Leave empty for performance, or AES256 for increased security).
    • sse_kms_key_id> (Leave empty).
  3. Validate and Exit: Validate your choices and exit the configuration.
  4. Manage Data: Use rclone commands to manage your data.
    • rclone lsd <config_name>:(list buckets/folders).
    • rclone copy /path/to/local/folder qarnot-storage:target-bucket(copy folder to remote)
    • rclone copy qarnot-storage:target-bucket /path/to/local/folder (copy remote folder to local)
    • rclone mount qarnot-storage:my-bucket /path/to/mountpoint (mount a Qarnot bucket/folder to the /path/to/mountpoint mointpoint)
  5. Verify Uploads: Check your data on Tasq.

‍

Use rclone (Web UI) to Connect to Qarnot

  1. Launch Web UI: Type rclone rcd --rc-web-gui to open the rclone Web UI in your browser.
  2. Configure Access:
    • Go to "Config."
    • Click "New config."
    • Remote Name: Enter a name for your Qarnot storage (e.g., "qarnot-storage").
    • Storage Type: Select "S3."
  3. Drive Setup:
    • Provider: Select "Ceph."
    • Access Key ID: Enter your Qarnot account email.
    • Secret Access Key: Enter your Qarnot API token.
    • Endpoint: Enter https://storage.qarnot.com.
  4. Next: Click "Next" to save the configuration.
  5. Manage Data: Go to "Explorer" to manage your data.

Optional: Mount Folder (linux & macos only)

  1. Go to "Mount."
  2. Click "New mount."
  3. FS: Select your Qarnot storage remote.
  4. Mount Point: Select the local directory where you want to mount the storage.
  5. Create: Click "Create."

You can now access your Qarnot storage through your file explorer.

‍