Command: uploadRequired Options: -a --attribute, -t --tag, -z --zone, -s --source-file, --zip, -td --thread, -o --output path Usage: Upload files/folders to the given project, with optional tags and attributes, to the target folder. Output: Prerequisite: User login is required, otherwise error is returned: "The current login session is invalid. Please login to continue." Project must be accessible to the user, otherwise, error is returned: "Project Code not found in list. Please verify and try again." Project path must include destination folder type (i.e., shared vs. users) If uploading using -t --tag, a JSON listing all tags is required. There is a limit of 10 tags per upload.
⚠️Important: Operating Systems use a different syntax to designate a file path on a local device. Windows OS uses a backslash between directories (e.g. C:\folder\subfolder\file.txt) while Linux and Mac OS use forward slashes between directories (e.g. /folder/subfolder/file.txt). Most examples in this page will use Linux and Mac OS style paths unless otherwise specified, please adjust if you are using Windows. | Code Block |
|---|
$ pilotcli file upload --help
Usage: pilotcli file upload [OPTIONS] OBJECT_PATH [FILES]...
Upload files/folders to a given Project path (eg.
<project>/users/<user>/<path>).
Options:
-a, --attribute FILENAME Add attributes to the file using a File Attribute
Template.
-t, --tag FILENAME Add tags to the file using a Tag file.
-z, --zone TEXT Target Zone (i.e., core/greenroom). [default:
greenroom]
-s, --source-file TEXT Project file path for identifying a source file
when creating an upstream file lineage node.
Source files must exist in the Core zone.
--zip Upload folder as a compressed zip file.
-td, --thread INTEGER The number of threads for uploading a file.
[default: 1]
-o, --output-path TEXT The output path for the manifest file of resumable
upload log. [default:
./resumable_upload_log.json]
--help Show this message and exit. |
Examples(1) Upload file to a Shared Folder in the Green Room (on Windows)| Code Block |
|---|
$ pilotcli file upload indoctestproject/shared/IMG C:\Users\name\Documents\test.csv
Starting upload of: C:\Users\name\Documents\test.csv
Checking for file duplication...
Pre-upload complete.
Uploading C:\Users\name\Documents\test.csv: |██████████████████████████████ 100% 00:00
Upload Time: 19.39s for 1 files
All uploading jobs have finished. |
(2) Upload file to a Shared Folder in the Green Room (on Linux/Mac OS)Select a file and upload it to a project Green Room folder. Command structure: pilotcli file upload <project name> <project code>/<folder type> (users or shared)/<folder <file name> <file input location>
| Code Block |
|---|
$ pilotcli file upload indoctestproject/shared/admin ~/Documents/test.csv
Starting upload of: ~/Documents/test.csv
Checking for file duplication...
Pre-upload complete.
Uploading ~/Documents/test.csv: |██████████████████████████████ 100% 00:00
Upload Time: 19.39s for 1 files
All uploading jobs have finished. |
(3) Upload file to a Shared Folder in the CoreSelect a file and upload to a project Core folder. Please recall that this command can only be run within a virtual machine that is deployed in the project core. If you are running the pilotcli tool from your local machine, or from a virtual machine deployed in the greenroom zone, you will run into permission issues. For more details on project zone restrictions, please see: https://indocpilot.atlassian.net/servicedesk/customer/portal/1/topic/bc65e100-23e5-4b50-9a93-3ccb5db1049f/article/1082927 . Command structure: pilotcli file upload <project name>code>/<folder type> (users or shared)/<file name> <file input location>-z <zone> (i.e., core or greenroom)
| Code Block |
|---|
$ pilotcli file upload indoctestproject/shared/IMG ~/Documents/test.csv -z core
You are about to transfer data directly to the Core! In accordance with the Terms of Use, please confirm that you
have made your best efforts to pseudonymize or anonymize the data and that you have the legal authority to transfer and make this
data available for dissemination and use within the Platform.To review the Terms of Use, please the Portal. If you
need to process the data to remove sensitive identifiers, please cancel this transfer and upload the data to the Green Room to
perform these actions.
To cancel this transfer, enter [n/No]
To confirm and proceed with the data transfer, enter [y/Yes]
[y/N]: y
Starting upload of: ~/Documents/test.csv
Checking for file duplication...
Pre-upload complete.
uploader: admin container_id: 18061 total_size: 1048576 total_chunks: 1 uniq_identifier: fa807a0f-ceb0-4fff-9c9e-563bdcbb309f-1627498851 resumable_relative_path admin/
Upload job is finalizing, please wait...
Upload job complete.
All files uploaded successfully. |
(4) Upload folder to a Shared Folder in the Green Room| Code Block |
|---|
$ pilotcli file upload indoctestproject/shared/IMG ~/Documents/test_folder
Starting upload of: ~/Documents/test_folder
Checking for file duplication...
Pre-upload complete.
Uploading example_file: |██████████████████████████████ 100% 00:00
Upload Time: 17.53s for 1 files
All uploading jobs have finished. |
(5) Upload file to an existing folder in a Shared Folder in the Green RoomSpecify a target folder to upload files by inputting folder name after the Project code separated by the '/'. Command structure: pilotcli file upload <project name>code>/<folder type> (users or shared)/<file name>/<sub-folder name> <file input location>
| Code Block |
|---|
$ pilotcli file upload indoctestproject/shared/IMG/test ~/Documents/testfile.zip
Checking for file duplication...
Pre-upload complete.
Uploading ~/Documents/testfile.zip: |██████████████████████████████ 100% 00:00
Upload Time: 19.39s for 1 files
All uploading jobs have finished. |
(6) Upload folder as zip to a Shared Folder in the Green Room*Note: When using this option, pilotcli must have permission to create zip file in the directory When uploading a folder through pilotcli, use the --zip option to compress the folder as a zip file and upload to the Shared folder. This option will first compress the folder as a zip file in the same directory as the folder. After the zip file is uploaded, the compressed zip will be removed from the directory. Command structure: pilotcli file upload <project name>code>/<folder type> (users or shared)/<folder name> <folder input location>--zip
| Code Block |
|---|
$ pilotcli file upload indoctestproject/shared/IMG ~/Documents/test_folder --zip
Starting upload of: ~/Documents/test_folder
Started zipping files.
Checking for file duplication...
Pre-upload complete.
Uploading test_folder.zip: |██████████████████████████████ 100% 00:00
Upload Time: 18.01s for 1 files
All uploading jobs have finished. |
(7) Upload folder to a Shared Folder in the Green Room and create a new folderWhen using pilotcli to upload a file/folder, you can create a new target folder within an existing folder by adding it to the destination pathway. If the command line interface detects that the specified target folder does not yet exist, it will prompt to confirm creation of the new folder. Confirm by entering 'y' (yes), then pilotcli will upload the file to the new created folder, otherwise the current uploading task is stopped. Top level shared and users folder creation is not supported. Only the creation of new sub-folders within existing shared/users folders is supported. Command structure: pilotcli file upload <project name>code>/<folder type> (users or shared)/<existing folder name>/<new folder name> <file input location>
| Code Block |
|---|
$ pilotcli file upload indoctestproject/shared/IMG/new_folder ~/Documents/test.csv
Target folder does not exist. Would you like to create a new folder? [y/N]: y
Starting upload of: ~/Documents/test.csv
Pre-upload complete.
Uploading example_file: |██████████████████████████████ 100% 00:00
Upload Time: 12.94s for 1 files
All uploading jobs have finished. |
To upload a file with attributes, user must first export an attribute template as a JSON file and fill the template with the desired values. When uploading files, use the -a/--attribute option to identify the attribute template JSON file. The Pilot Command Line Interface is capable of validating attribute template. Whenever a template is invalid, including missing non-optional fields, non-existing choice in multiple choice attribute, exceeding length text field and so on, pilotcli will raise error to user and user must update the attribute template and upload again. Optionally, use the -t/--tag option to include tags. Tags must be listed in a JSON file. Maximum 10 tags per file. Only files can be uploaded with attributes / tags. Attempting to upload a Folder with -t/--tag or -a/--attribute will result in the following error message:
”Tagging and manifest attaching are not support for folder type.” Command structure: pilotcli file upload <project name>code>/<folder type> (users or shared)/<new folder name> <file input location> -t <tag json> -a <attribute json>
| Code Block |
|---|
$ pilotcli file upload indoctestproject/shared/IMG ~/Documents/test.csv -t tags.json -a IMG_CoreMetadata.json
File attribute validation passed: True
Starting upload of: ./sample_folder/sample_file6
Pre-upload complete.
Uploading sample_file6: |██████████████████████████████ 100% 00:00
Upload Time: 14.96s for 1 files
Attribute attached
All uploading jobs have finished. |
(9) Upload file to a Shared Folder in the Core and build lineage with an existing fileUsers who have access to the project Core zone can upload a file to a project Core folder and build lineage with an existing file. Please recall that this command can only be run within a virtual machine that is deployed in the project Core. If you are running the pilotcli tool from your local machine, or from a virtual machine deployed in the greenroom zone, you will run into permission issues. For more details on project zone restrictions, please see: https://indocpilot.atlassian.net/servicedesk/customer/portal/1/topic/bc65e100-23e5-4b50-9a93-3ccb5db1049f/article/1082927 . To build lineage, user needs to upload a file to the Core zone, then identify the source file with -s/--source-file option. After that, the file will be uploaded to the project core folder as Downstream and build a lineage with identified source file as Upstream. Please note that when listing the source file path, the project code should not be included. See example below for more details. Command structure: pilotcli file upload <project name>code>/<folder type> (users or shared)/<folder name> <file input location> -z <zone> (i.e., core or greenroom)-s <source folder type>(users or shared)/<source folder name>/<source file name>
| Code Block |
|---|
indocdemo@indoctestproject-demo:~$ pilotcli file upload indoctestproject/users/swangadmin/testfolder0501 ./participants.tsv -s users/swangadmin/dataset_description.json -z core
You are about to transfer data directly to the PILOT Core! In accordance with the PILOT Terms of Use, please confirm that you have made your best
efforts to pseudonymize or anonymize the data and that you have the legal authority to transfer and make this data available for dissemination and use
within the PILOT .If you need to process the data to remove sensitive identifiers, please cancel this transfer and upload the data to the Green Room to
perform these actions.
To cancel this transfer, enter [n/No]
To confirm and proceed with the data transfer, enter [y/Yes]
[y/N]: y
The manifest file of folder ./resumable_upload_log.json already exist. Do you want to overwrite the existing manifest file? [y/N]: y
Target folder does not exist. Would you like to create a new folder? [y/N]: y
Starting upload of: ./participants.tsv
Pre-upload complete.
Uploading participants.tsv: |██████████████████████████████ 100% 00:00
Upload Time: 2.66s for 1 files
All uploading jobs have finished.
|
(10) Upload file to a Shared Folder in the Green Room using threadsUsing threads allows for the concurrent uploading of multiple files at the same time. Specify -td <thread_number> to indicate the number of threads. By default, the number of threads is set to 1. The max number of thread depends on your machine. The recommend thread number is 3. There will no UI difference when using threads. Command structure: pilotcli file upload <project name>code>/<folder type> (users or shared)/<folder name> <file input location> -td <thread count>
| Code Block |
|---|
$ pilotcli file upload indoctestproject/shared/IMG ~/Documents/test.csv -td 3
Target folder does not exist. Would you like to create a new folder? [y/N]: y
Starting upload of: ~/Documents/test.csv
Pre-upload complete.
Uploading ~/Documents/test.csv: |██████████████████████████████ 100% 00:00
Upload Time: 14.93s for 1 files
All uploading jobs have finished. |
(11) Upload file to a Shared Folder in the Green Room using a custom name for the resumable upload file JSON Upload has an option --output-path ,-o to indicate the name of the resumable upload file JSON. All uploads generate a resumable upload file (resumable_upload_log.json) to use when resuming a stopped upload. Command structure: pilotcli file upload <project name>code>/<folder type> (users or shared)/<folder name> <file input location> --output-path <manifest file name and path>
| Code Block |
|---|
$ pilotcli file upload indoctestproject/shared/IMG ~/Documents/test.csv --output-path ~/Documents/newmanifestfile.json
Starting upload of: ~/Documents/test.csv
Pre-upload complete.
Uploading ~/Documents/test.csv: |██████████████████████████████ 100% 00:00
Upload Time: 14.93s for 1 files
All uploading jobs have finished. |
(12) Upload folder to a Shared Folder in the Green Room and merge folder contentsIf the folder exists in the destination repository, the contents will be checked for duplicated files (case insensitive). Any new files can be uploaded by indicating that you’d like to skip duplicates and continue uploading. Command structure: pilotcli file upload <project name>code>/<folder type> (users or shared)/<folder name>/<sub folder name> <folder input location>
| Code Block |
|---|
$ pilotcli file upload indoctestproject/shared/IMG/test_folder ~/Documents/test_folder
Starting upload of: ~/Documents/test_folder
Checking for file duplication...
File duplication check complete.
Some of the selected files cannot be uploaded.
The following files already exist in the upload destination:
test_file_01
Do you want to cancel the upload [N] or skip duplicates and continue uploading [y]? [y/N]: y
Pre-upload complete.
Uploading test_file_02: |██████████████████████████████ 100% 00:00
Upload Time: 29.86s for 1 files
All uploading jobs have finished. |
|