File commands for the pilotcli command line tool.

File Commands: pilotcli file

Help

$ pilotcli file --help
Usage: pilotcli_windows file [OPTIONS] COMMAND [ARGS]...

Options:
  --help  Show this message and exit.

Commands:
  attribute-export  Export attribute template from a given Project.
  attribute-list    List attribute templates of a given Project.
  download          Download files/folders from a given Project/folder/file...
  list              List files and folders inside a given Project/folder.
  metadata          Download metadata file of a given file in target zone.
  move              Move/Rename files/folders to a given Project path.
  resume            Resume the upload process with a resumable upload log.
  trash             Move files/folders to trash bin.
  upload            Upload files/folders to a given Project path (eg.

Command: attribute-export

Options: -p --project-code, -n --attribute-name
Usage: Export an attribute from given Project by attribute name
Output:
- (1) Attribute Template JSON file
- (2) Attribute Definition JSON file
Prerequisites:
- User login is required, otherwise error is returned: "The current login session is invalid. Please login to continue."
- Project must be accessible to the user, otherwise, error is returned: "Project Code not found in list. Please verify and try again."
- Requires the Project Code (retrieved via pilotcli project list).
- Requires that the Attribute Template exists. If Attribute Template does not exist or is spelled incorrectly, the following error is returned: "Attribute 'sample-template' not found in Project. Please verify and try again."
- pilotcli must have permission to write file in the directory

$ pilotcli file attribute-export --help
Usage: pilotcli_linux file attribute-export [OPTIONS]

  Export attribute template from a given Project.

Options:
  -p, --project-code TEXT    Project Code
  -n, --attribute-name TEXT  Attribute Template Name
  --help                     Show this message and exit.

Examples

(1) Export attributes from a given Project by attribute name:

Directly export an attribute by name from a given project
Denoting the project options are not required in the command explicitly, however if they are not included you will be prompted to enter them
Command structure: pilotcli file attribute-export
- example: pilotcli file attribute-export

$ pilotcli file attribute-export
ProjectCode: <Enter-Project-Code>
AttributeName: <Enter-Attribute-Template-Name>

Attribute-Template-Name
--------------------------------------------------------------------------
|    Attribute Name    |       Type      |        Value         | Optional |
--------------------------------------------------------------------------
|        attr1         | multiple_choice |    a1,a2,a3,a4,a5    |  False   |
--------------------------------------------------------------------------
|        attr2         |       text      |         None         |  False   |
--------------------------------------------------------------------------
|        attr3         | multiple_choice |    t1,t2,t3,t4,t5    |   True   |
--------------------------------------------------------------------------
Template saved successfully: Project-Code_Attribute-Template-Name_template.json
Attribute definition saved successfully: Project-Code_Attribute-Template-Name_definition.json

(2) Export attributes from a given Project by attribute name with options:

Directly export an attribute by name from a given project, including the options in the command itself.
Command structure: pilotcli file attribute-export -p <project code> -n <attribute template name>
- example: pilotcli file attribute-export -p IndocTestProject -n IMG_CoreMetadata

$ pilotcli file attribute-export -p Project-Code -n Attribute-Template-Name

Attribute-Template-Name
--------------------------------------------------------------------------
|    Attribute Name    |       Type      |        Value         | Optional |
--------------------------------------------------------------------------
|        attr1         | multiple_choice |    a1,a2,a3,a4,a5    |  False   |
--------------------------------------------------------------------------
|        attr2         |       text      |         None         |  False   |
--------------------------------------------------------------------------
|        attr3         | multiple_choice |    t1,t2,t3,t4,t5    |   True   |
--------------------------------------------------------------------------
Template saved successfully: Project-Code_Attribute-Template-Name_template.json
Attribute definition saved successfully: Project-Code_Attribute-Template-Name_definition.json

Command: attribute-list

Options: -p --project-code
Usage: List all attributes from given Project
Prerequisite:
- User login is required, otherwise error is returned: ‘The current login session is invalid. Please login to continue.’
- Requires the Project code (retrieved via pilotcli project list).

$ pilotcli file attribute-list --help
Usage: pilotcli file attribute-list [OPTIONS]

  List attribute templates of a given Project.

Options:
  -p, --project-code TEXT  Project Code
  --help                   Show this message and exit.

Examples

(1) List attributes from a given Project:

Generate a list of attributes in any given project.
Denoting the project options are not required in the command explicitly, however if they are not included you will be prompted to enter them
Command structure: pilotcli file attribute-list
- example: pilotcli file attribute-list

$ pilotcli file attribute-list
ProjectCode: <Enter-Project-Code>

Attribute-Template-Name
--------------------------------------------------------------------------
|    Attribute Name    |       Type      |        Value         | Optional |
--------------------------------------------------------------------------
|        attr1         | multiple_choice |    a1,a2,a3,a4,a5    |  False   |
--------------------------------------------------------------------------
|        attr2         |       text      |         None         |  False   |
--------------------------------------------------------------------------
|        attr3         | single_choice |    t1,t2,t3,t4,t5    |   True   |
--------------------------------------------------------------------------

Attribute-Template-Name2
--------------------------------------------------------------------------
|    Attribute Name    |       Type      |        Value         | Optional |
--------------------------------------------------------------------------
|        attr1         | multiple_choice |     a1,a2,a3,a4      |   True   |
--------------------------------------------------------------------------
|        attr2         |       text      |         None         |  False   |
--------------------------------------------------------------------------
|        attr3         | single_choice |       t1,t2,t3       |   True   |
--------------------------------------------------------------------------
All Attributes fetched successfully.

(2) List attributes from a given Project with options:

Directly generate a list of attributes from a given project, including the options in the command itself.
Command structure: pilotcli file attribute-list -p <project code>
- example: pilotcli file attribute-list -p IndocTestProject

$ pilotcli file attribute-list -p <Project-Code>

Attribute-Template-Name
--------------------------------------------------------------------------
|    Attribute Name    |       Type      |        Value         | Optional |
--------------------------------------------------------------------------
|        attr1         | multiple_choice |    a1,a2,a3,a4,a5    |  False   |
--------------------------------------------------------------------------
|        attr2         |       text      |         None         |  False   |
--------------------------------------------------------------------------
|        attr3         | single_choice |    t1,t2,t3,t4,t5    |   True   |
--------------------------------------------------------------------------

Attribute-Template-Name2
--------------------------------------------------------------------------
|    Attribute Name    |       Type      |        Value         | Optional |
--------------------------------------------------------------------------
|        attr1         | multiple_choice |     a1,a2,a3,a4      |   True   |
--------------------------------------------------------------------------
|        attr2         |       text      |         None         |  False   |
--------------------------------------------------------------------------
|        attr3         | single_choice |       t1,t2,t3       |   True   |
--------------------------------------------------------------------------
All Attributes fetched successfully.

Command: download

Usage: Download a file/folder from a Project using a full path: project name/folder type (users vs. shared)/folder name/file or folder name (i.e., indoctestproject/users/username/folder1/file.csv)
Options: -z --zone, -i --geid, --zip
- To look up a file’s geid, or unique file ID, open the Project File Explorer in the Portal and inspect the File Properties for the ID.
Prerequisite:
- User login is required, otherwise error is returned: "The current login session is invalid. Please login to continue."
- File/folder must exist, otherwise error is returned: “File does not exist or source file provided is invalid: folder_name/file_name.”
- If user has no access to the file/folder on the Portal, an error is returned: “Permission denied. Please verify that your role in the Project has permission to perform this action.”

Please be aware of the applicable zone restrictions depending on where you are using the pilotcli: Zone Restrictions with pilotcli

$ pilotcli file download --help
Usage: pilotcli_linux file download [OPTIONS] [PATHS]... OUTPUT_PATH

  Download files/folders from a given Project/folder/file in core zone.

Options:
  -z, --zone TEXT  Target Zone (i.e., core/greenroom).
  --zip            Download files as a zip.
  -i, --geid       Enable downloading by geid.
  --help           Show this message and exit.

Examples

(1) Download file/folder from Green Room

Downloading a file/folder from Green Room with download command requires the environment variable to be configured, and pilotcli will prevent upload of duplicate file names (case insensitive). If user inputs multiple file/folder names, these files or folders will be downloaded one by one to the given directory.
Command structure: pilotcli file download <project name>/<folder type> (users or shared)/<folder name>/<file or folder name> <file output location>
- example: pilotcli file download indoctestproject/shared/IMG/test.csv ~/Documents

$ pilotcli file download indoctestproject/users/admin/sample_file2 cli/admin/sample_file7 ~/Downloads
Preparing status: READY_FOR_DOWNLOADING
/home/uname/Downloads/sample_file2 already exist, file will be saved as /home/uname/Downloads/sample_file2 (1)
start downloading...
Downloading sample_file2 (1) |██████████████████████████████ 100% 00:00
File has been downloaded successfully and saved to: /home/uname/Downloads/sample_file2 (1)
Preparing status: READY_FOR_DOWNLOADING
start downloading...
Downloading sample_file7 |██████████████████████████████ 100% 00:00
File has been downloaded successfully and saved to: /home/uname/Downloads/sample_file7

(2) Download file/folder from Core

Use the download command to download a file/folder from Core with option -z or --zone.
Command structure: pilotcli file download <project name>/<folder type> (users or shared)/<folder name>/<file or folder name> -z <zone> (i.e., greenroom or core) <file output location>
- example: pilotcli file download indoctestproject/shared/IMG/test.csv -z core ~/Documents

$ pilotcli file download testproject/shared/admin/sample_file2 -z core ~/Downloads/
Preparing status: READY_FOR_DOWNLOADING
start downloading...
Downloading sample_file2 |██████████████████████████████ 100% 00:00
File has been downloaded successfully and saved to: /home/uname/Downloads/sample_file2

(3) Download multiple files/folders as a compressed zip

Use the download command to download multiple files/folders as a compressed zip file with option --zip.
Command structure: pilotcli file download <project name>/<folder type> (users or shared)/<folder name>/<file or folder name> -z <zone> (i.e., greenroom or core) <file output location> --zip
- example: pilotcli file download indoctestproject/shared/IMG/test.csv -z core ~/Documents
  --zip

$ pilotcli file download testproject/shared/admin/sample_file2 cli/admin/sample_file7 -z core ~/Downloads/ --zip
Preparing downloading
Preparing status: READY_FOR_DOWNLOADING
start downloading...
Downloading cli_1632340658.768384.zip |██████████████████████████████ 100% 00:00
File has been downloaded successfully and saved to: /home/uname/Downloads/cli_1632340658.768384.zip

(4) Download file/folder by `geid`

Use the download command to download file(s)/folder(s) by geid with option -i --geid. The option -z --zone is not required when downloading by geid.
Command structure: pilotcli file download -i <geid from file 1> <geid from file 2> <file output location>
- example pilotcli file download -i 6d89afd4-f86e-4df7-b6c5-01a9af4e828e-1627499957 7e8e2bac-97c6-4738-bc08-12139c9ba225-1627498854 ~/Documents

$ pilotcli file download -i 6d89afd4-f86e-4df7-b6c5-01a9af4e828e-1627499957 7e8e2bac-97c6-4738-bc08-12139c9ba225-1627498854  ~/Downloads/
Preparing status: READY_FOR_DOWNLOADING
start downloading...
Downloading sample_file7 |██████████████████████████████ 100% 00:00
File has been downloaded successfully and saved to: /home/uname/Downloads/sample_file7
Preparing status: READY_FOR_DOWNLOADING
start downloading...
Downloading sample_file2 |██████████████████████████████ 100% 00:00
File has been downloaded successfully and saved to: /home/uname/Downloads/sample_file2

If the --zip option is used then command interface will automatically group input geid by the project, zone and then download as different zip files.
Command structure: pilotcli file download -i <geid from file 1> <geid from file 2> <file output location> --zip
- example: pilotcli file download -i 6d89afd4-f86e-4df7-b6c5-01a9af4e828e-1627499957 7e8e2bac-97c6-4738-bc08-12139c9ba225-1627498854 ~/Documents --zip

$ pilotcli file download -i --zip 6d89afd4-f86e-4df7-b6c5-01a9af4e828e-1627499957 7e8e2bac-97c6-4738-bc08-12139c9ba225-1627498854 228ec18b-6775-443d-a69e-6c491b240b98-1627498776 25f0ede8-859e-4320-8e53-83921aa9a6be-1627499276 ~/Downloads/ --zip
Preparing downloading
Preparing status: READY_FOR_DOWNLOADING
start downloading...
Downloading cli_1632341269.0496638.zip |██████████████████████████████ 100% 00:00
File has been downloaded successfully and saved to: /home/uname/Downloads/cli_1632341269.0496638.zip
Preparing status: READY_FOR_DOWNLOADING
start downloading...
Downloading cli_1632341274.5802007.zip |██████████████████████████████ 100% 00:00
File has been downloaded successfully and saved to: /home/uname/Downloads/cli_1632341274.5802007.zip

Command: list

Options: -z --zone
Usage: List files and folders in the directory. Folders will be displayed in blue and files will be displayed in white.
Prerequisite:
- User login is required, otherwise error is returned: "The current login session is invalid. Please login to continue."
- Project must be accessible to the user, otherwise, error is returned: "Project Code not found in list. Please verify and try again."
- Requires the Project code (retrieved via pilotcli project list) and path.
- Requires indicating whether the folder is a shared folder or a user folder in the file path. Not denoting this will return only two options shared users

By default, only the first ten elements will be displayed. If you have access to more than ten files/folders in a certain Project, you will be prompted to use your arrow keys to select one of two options to either view the next ten items (next page), or to exit (exit) to return to the command line. The selected option is indicated by two arrows and can be confirmed by hitting enter:

What do you want? (Use arrow keys)
 » next page
   exit

$ pilotcli file list --help
Usage: pilotcli_linux file list [OPTIONS] PATHS

  List files and folders inside a given Project/folder.

Options:
  -z, --zone TEXT      Target Zone (i.e., core/greenroom).  [default:
                       greenroom]

  --page INTEGER       The page to be listed  [default: 0]
  --page-size INTEGER  number of objects per page  [default: 10]
  -d, --detached       whether run in detached mode
  --help               Show this message and exit.

Examples

(1) List files/folders in a Green Room user folder

List all files/folders in a Green Room folder
-z is not required, however you must denote whether the folder is a shared folder or a user folder in the path in order to return the file/folder list
Command structure: pilotcli file list <project name>/<folder type> (users or shared)/<folder name>
- example: pilotcli file list indoctestproject/shared/IMG

$ pilotcli file list indoctestproject/users/admin
test_folder1 test_folder2 file1 file2 file3

(2) List files/folders in a Core shared folder

List all files/folders in a Core folder
Command structure: pilotcli file list <project name>/<folder type> (users or shared)/<folder name> -z <zone> (i.e., greenroom or core)
- example: pilotcli file list indoctestproject/shared/IMG -z core

$ pilotcli file list indoctestproject/shared/admin -z core
folder1 folder2 test_folder.zip file.txt sample.txt large-file.json

(3) List files/folders in Trash Bin from the Green Room

List all files/folders in the Trash Bin from the Green Room
Non-administrator users with access to the Trash Bin can only view the files they have moved to trash. Platform and Project Administrators can view the files in the Trash Bin from all Project members.
Command structure: pilotcli file list <project name>/trash -z greenroom
- example: pilotcli file list indoctestproject/trash -z greenroom

$ pilotcli file list indoctestproject/trash -z greenroom
folder1 folder2 test_folder.zip file.txt sample.txt large-file.json

(3) List files/folders in Trash Bin from the Core

List all files/folders in the Trash Bin from the core
Non-administrator users with access to the Trash Bin can only view the files they have moved to trash. Platform and Project Administrators can view the files in the Trash Bin from all Project members.
Command structure: pilotcli file list <project name>/trash -z core
- example: pilotcli file list indoctestproject/trash -z core

$ pilotcli file list indoctestproject/trash -z core
folder1 folder2 test_folder.zip file.txt sample.txt large-file.json

Command: metadata

Usage: Download a file’s metadata including properties, attributes, and tags.
Options: -z --zone, -g --general, -a --attribute, -t --tag

Output:

(1) attribute.json - a file containing the file’s attributes

{
    "<template_name>": {
        "<attribute_1>": "<value_1>",
    }
}

(2) general.json - a file containing the file’s general metadata

#sample general.json file
{
    "id": "40219ea6-ef6a-436a-8458-2d654b27e239",
    "parent": "ff009692-537b-45da-b74a-a4255adf5eec",
    "parent_path": "users/blasalandra",
    "restore_path": null,
    "status": "ACTIVE",
    "type": "file",
    "zone": 0,
    "name": "TestFile.pdf",
    "size": 128000,
    "owner": "blasalandra",
    "container_code": "indoctestproject",
    "container_type": "project",
    "created_time": "2024-03-20 17:28:58.365001",
    "last_updated_time": "2024-03-20 17:29:00.690349",
    "storage": {
        "id": "2db51316-2d7d-40a9-96a8-dfe05ca2ea2a",
        "version": null,
        "upload_id": "a353b5de-53a2-46e6-9ca8-be3dd34e5cba"
    },
    "favourite": false
}

(3) tag.json
- ```
["tag1", "tag2", "tag3"]
```

Prerequisites:
- User login is required, otherwise error is returned: "The current login session is invalid. Please login to continue."
- File must exist, otherwise error is returned: “File does not exist or source file provided is invalid: folder_name/file_name.”
- If user has no access to the file/folder on the Portal, an error is returned: “Permission denied. Please verify that your role in the Project has permission to perform this action.”

$ pilotcli file metadata --help
Usage: pilotcli_linux file metadata [OPTIONS] FILE_PATH

  Download metadata file of a given file in target zone.

Options:
  -z, --zone TEXT       Target Zone (i.e., core/greenroom).  [required]
  -g, --general TEXT    The location of general metadata file  [required]
  -a, --attribute TEXT  The location of attribute metadata file  [required]
  -t, --tag TEXT        The location of tag metadata file  [required]
  --help                Show this message and exit.

(1) Download a file with general metadata, attributes, and tags.

Download file metadata in the form of: attribute.json, general.json, tags.json
Command structure: pilotcli file metadata -g . -a . -t . -z <zone> (i.e., greenroom or core) <project name>/<folder type> (users or shared)/<folder name>/<file or folder name>
- example: pilotcli file metadata -g . -a . -t . -z core indoctestproject/shared/IMG/testfile.csv

$ pilotcli file metadata -g . -a . -t . -z core projectcode/users/username/folder/mock_csvfile.csv
Metadata download complete.

Command: move

Options: -z --zone
Usage: Move/rename files/folders to a given Project Path within the same project
Prerequisite:
- User login is required, otherwise error is returned: "The current login session is invalid. Please login to continue."
- Project must be accessible to the user, otherwise, error is returned: "Project Code not found in list. Please verify and try again."
- Project path must include the project folder type (i.e., shared or users).
- When moving a file to the Project Core, user must have access to the Core (Project Contributors cannot move files/folders to the Project Core).

This command cannot be used to generate a new folder within a user or shared top-level folder. To create a new folder, please see the Folder Commands - pilotcli folder.

$ pilotcli file move --help
Usage: pilotcli_linux file move [OPTIONS] SRC_ITEM_PATH DEST_ITEM_PATH

  Move/Rename files/folders to a given Project path.

Options:
  -z, --zone TEXT  Target Zone (i.e., core/greenroom).  [required]
  --help           Show this message and exit.

Examples

(1) Move a file from one folder to another within a Green Room Project

Move a file from one folder to another folder within that same project in the project Greenroom.
When moving/renamine a file within the project Greenroom, -z is not required.
Command structure: pilotcli file move <project code>/<origin folder type> (users or shared)/<origin folder name>/<file or folder name> <project code>/<destination folder type> (users or shared)/<destination folder name>
- example: pilotcli file move indoctestproject/shared/IMG/test.csv indoctestproject/users/blasalandra/

$ pilotcli file move indoctestproject/users/blasalandra/test indoctestproject/shared/SharedFolder
Successfully moved indoctestproject/users/blasalandra/test to indoctestproject/shared/SharedFolder

(2) Move a file from one folder to another within a Core Project

Move a file from one folder to another folder within the same project in the project core zone.
Command structure: pilotcli file move <project code>/<origin folder type> (users or shared)/<origin folder name>/<file or folder name> <project code>/<destination folder type> (users or shared)/<destination folder name> -z <zone> (i.e., greenroom or core)
- example: pilotcli file move indoctestproject/shared/IMG/test.csv indoctestproject/users/blasalandra -z core

$ pilotcli file move indoctestproject/users/blasalandra/test indoctestproject/shared/SharedFolderCore -z core
Successfully moved indoctestproject/users/blasalandra/test to indoctestproject/shared/SharedFolderCore/test

(3) Rename a file in a Green Room folder

Rename a file from the command line using the move command.
Command structure: pilotcli file move<project code>/<folder type> (users or shared)/<folder name>/<original file or folder name> <project code>/<folder type> (users or shared)/<folder name>/ <new file or folder name>
- example: pilotcli file move indoctestproject/shared/IMG/test.csv indoctestproject/shared/IMG/newfile.csv

$ pilotcli file move indoctestproject/shared/test/testfile indoctestproject/shared/test/newfilename
Successfully moved indoctestproject/shared/test/testfile to indoctestproject/shared/test/newfilename

Command: resume

Options: -td --thread, -r --resumable-manifest
Usage: Upload unfinished file/folder uploading based on the resumable upload file (resumable_upload_log.json)
Prerequisite:
- User login is required, otherwise error is returned: "The current login session is invalid. Please login to continue."
- Project must be accessible to the user, otherwise, error is returned: "Project Code not found in list. Please verify and try again."
- Resumable upload manifest file is required, which is generated from using the file upload command (see example 10 above) [default: ./resumable_upload_log.json]

Please note incomplete uploads will be purged and removed from Pilot after 24 hours. Resumable files can also be managed or deleted from within the Pilot portal, please see this page for more information: Resuming Failed File Uploads

Please note, if the source or destination locations are altered (either renamed or removed) before running the resume upload command, resuming upload will fail and you will be prompted to re-start the upload utilizing the upload command.

$ pilotcli file resume --help
Usage: pilotcli file resume [OPTIONS]

  Resume the upload process with given manifest file.

Options:
  -td, --thread INTEGER          The number of thread for upload a file
                                 [default: 1]

  -r, --resumable-manifest TEXT  The manifest file for resumable upload
                                 [required]

  --help                         Show this message and exit.

Examples

(1) Resume the upload with resumable upload file (`resumable_upload_log.json)`

The resumable command uses the manifest file generated from file upload to resume all uploads
Command structure: pilotcli file resume -r <resumable manifest json file>
- example: pilotcli file resume -r ~/Documents/resumable_upload_log.json

$ pilotcli file resume -r  ./resumable_upload_log.json
Resume the upload for 10 files.
Resumable upload check complete.
Uploading aggregated.py:  |██████████████████████████████ 100% 00:00
Uploading __init__.py:  |██████████████████████████████ 100% 00:00
Uploading service_meta_class.py:  |██████████████████████████████ 100% 00:00
Uploading upload_form.py:  |██████████████████████████████ 100% 00:00
Uploading convert_type.py:  |██████████████████████████████ 100% 00:00
Uploading singleton.py:  |██████████████████████████████ 100% 00:00
Uploading __init__.py:  |██████████████████████████████ 100% 00:00
Uploading base_library.zip:  |██████████████████████████████ 100% 00:00         0% ?
Uploading EXE-00.toc:  |██████████████████████████████ 100% 00:00      
Uploading xref-pilotcli_test20230227_3.html:  |██████████████████████████████ 100% 00:00
Upload Time: 11.13s for 10 files

Command: upload

Required Options: -a --attribute, -t --tag, -z --zone, -s --source-file, --zip, -td --thread, -o --output path
Usage: Upload files/folders to the given project, with optional tags and attributes, to the target folder.
Output:
- resumable_upload_log.json file: contains information to resume upload if disrupted.
Prerequisite:
- User login is required, otherwise error is returned: "The current login session is invalid. Please login to continue."
- Project must be accessible to the user, otherwise, error is returned: "Project Code not found in list. Please verify and try again."
- Project path must include destination folder type (i.e., shared vs. users)
- If uploading using -t --tag, a JSON listing all tags is required. There is a limit of 10 tags per upload.

⚠️Important: Operating Systems use a different syntax to designate a file path on a local device. Windows OS uses a backslash between directories (e.g. C:\folder\subfolder\file.txt) while Linux and Mac OS use forward slashes between directories (e.g. /folder/subfolder/file.txt). Most examples in this page will use Linux and Mac OS style paths unless otherwise specified, please adjust if you are using Windows.

Please be aware of the applicable zone restrictions depending on where you are using the pilotcli: Zone Restrictions with pilotcli

$ pilotcli file upload --help
Usage: pilotcli file upload [OPTIONS] OBJECT_PATH [FILES]...

  Upload files/folders to a given Project path (eg.
  <project>/users/<user>/<path>).

Options:
  -a, --attribute FILENAME  Add attributes to the file using a File Attribute
                            Template.

  -t, --tag FILENAME        Add tags to the file using a Tag file.
  -z, --zone TEXT           Target Zone (i.e., core/greenroom).  [default:
                            greenroom]

  -s, --source-file TEXT    Project file path for identifying a source file
                            when creating an upstream file lineage node.
                            Source files must exist in the Core zone.

  --zip                     Upload folder as a compressed zip file.
  -td, --thread INTEGER     The number of threads for uploading a file.
                            [default: 1]

  -o, --output-path TEXT    The output path for the manifest file of resumable
                            upload log.  [default:
                            ./resumable_upload_log.json]

  --help                    Show this message and exit.

Examples

(1) Upload file to a Shared Folder in the Green Room (on Windows)

Select a file and upload it to a project Green Room folder.
Command structure: pilotcli file upload <project name>/<folder type> (users or shared)/<folder name> <file input location>
- example: pilotcli file upload indoctestproject/shared/IMG C:\Users\name\Documents\test.csv

$ pilotcli file upload indoctestproject/shared/IMG C:\Users\name\Documents\test.csv 
Starting upload of: C:\Users\name\Documents\test.csv
Checking for file duplication...
Pre-upload complete.
Uploading C:\Users\name\Documents\test.csv:  |██████████████████████████████ 100% 00:00
Upload Time: 19.39s for 1 files
All uploading jobs have finished.

(2) Upload file to a Shared Folder in the Green Room (on Linux/Mac OS)

Select a file and upload it to a project Green Room folder.
Command structure: pilotcli file upload <project name>/<folder type> (users or shared)/<folder name> <file input location>
- example: pilotcli file upload indoctestproject/shared/IMG ~/Documents/test.csv

$ pilotcli file upload indoctestproject/shared/admin ~/Documents/test.csv 
Starting upload of: ~/Documents/test.csv
Checking for file duplication...
Pre-upload complete.
Uploading ~/Documents/test.csv:  |██████████████████████████████ 100% 00:00
Upload Time: 19.39s for 1 files
All uploading jobs have finished.

(3) Upload file to a Shared Folder in the Core

Select a file and upload to a project Core folder.
Please recall that this command can only be run within a virtual machine that is deployed in the project core. If you are running the pilotcli tool from your local machine, or from a virtual machine deployed in the greenroom zone, you will run into permission issues. For more details on project zone restrictions, please see: https://indocpilot.atlassian.net/servicedesk/customer/portal/1/topic/bc65e100-23e5-4b50-9a93-3ccb5db1049f/article/1082927 .
Command structure: pilotcli file upload <project name>/<folder type> (users or shared)/<file name> <file input location> -z <zone> (i.e., core or greenroom)
- example: pilotcli file upload indoctestproject/shared/IMG ~/Documents/test.csv -z core

$ pilotcli file upload indoctestproject/shared/IMG ~/Documents/test.csv  -z core
You are about to transfer data directly to the Core! In accordance with the Terms of Use, please confirm that you 
have made your best efforts to pseudonymize or anonymize the data and that you have the legal authority to transfer and make this 
data available for dissemination and use within the Platform.To review the Terms of Use, please the Portal. If you 
need to process the data to remove sensitive identifiers, please cancel this transfer and upload the data to the Green Room to 
perform these actions. 
To cancel this transfer, enter [n/No] 
To confirm and proceed with the data transfer, enter [y/Yes] 
  [y/N]: y
Starting upload of: ~/Documents/test.csv
Checking for file duplication...
Pre-upload complete.
uploader: admin container_id: 18061 total_size: 1048576 total_chunks: 1 uniq_identifier: fa807a0f-ceb0-4fff-9c9e-563bdcbb309f-1627498851 resumable_relative_path admin/
                                                                   
Upload job is finalizing, please wait...
Upload job complete.
All files uploaded successfully.

(4) Upload folder to a Shared Folder in the Green Room

Select a folder and upload to a project Green Room folder.
Command structure: pilotcli file upload <project name>/<folder type> (users or shared)/<folder name> <file input location>
- example: pilotcli file upload indoctestproject/shared/IMG ~/Documents/test_folder

$ pilotcli file upload indoctestproject/shared/IMG ~/Documents/test_folder 
Starting upload of: ~/Documents/test_folder
Checking for file duplication...
Pre-upload complete.
Uploading example_file:  |██████████████████████████████ 100% 00:00
Upload Time: 17.53s for 1 files
All uploading jobs have finished.

(5) Upload file to an existing folder in a Shared Folder in the Green Room

Specify a target folder to upload files by inputting folder name after the Project code separated by the '/'.
Command structure: pilotcli file upload <project name>/<folder type> (users or shared)/<file name>/<sub-folder name> <file input location>
- example: pilotcli file upload indoctestproject/shared/IMG/test ~/Documents/testfile.zip

$ pilotcli file upload indoctestproject/shared/IMG/test ~/Documents/testfile.zip 
Checking for file duplication...
Pre-upload complete.
Uploading ~/Documents/testfile.zip:  |██████████████████████████████ 100% 00:00
Upload Time: 19.39s for 1 files
All uploading jobs have finished.

(6) Upload folder as zip to a Shared Folder in the Green Room

*Note: When using this option, pilotcli must have permission to create zip file in the directory
When uploading a folder through pilotcli, use the --zip option to compress the folder as a zip file and upload to the Shared folder. This option will first compress the folder as a zip file in the same directory as the folder. After the zip file is uploaded, the compressed zip will be removed from the directory.
Command structure: pilotcli file upload <project name>/<folder type> (users or shared)/<folder name> <folder input location> --zip
- example: pilotcli file upload indoctestproject/shared/IMG ~/Documents/test_folder --zip

$ pilotcli file upload indoctestproject/shared/IMG ~/Documents/test_folder --zip
Starting upload of: ~/Documents/test_folder
Started zipping files.
Checking for file duplication...
Pre-upload complete.
Uploading test_folder.zip:  |██████████████████████████████ 100% 00:00
Upload Time: 18.01s for 1 files
All uploading jobs have finished.

(7) Upload folder to a Shared Folder in the Green Room and create a new folder

When using pilotcli to upload a file/folder, you can create a new target folder within an existing folder by adding it to the destination pathway. If the command line interface detects that the specified target folder does not yet exist, it will prompt to confirm creation of the new folder. Confirm by entering 'y' (yes), then pilotcli will upload the file to the new created folder, otherwise the current uploading task is stopped.
Top level shared and users folder creation is not supported. Only the creation of new sub-folders within existing shared/users folders is supported.
Command structure: pilotcli file upload <project name>/<folder type> (users or shared)/<existing folder name>/<new folder name> <file input location>
- example: pilotcli file upload indoctestproject/shared/IMG/NewFolder ~/Documents/test.csv

$ pilotcli file upload indoctestproject/shared/IMG/new_folder ~/Documents/test.csv 
Target folder does not exist. Would you like to create a new folder? [y/N]: y
Starting upload of: ~/Documents/test.csv
Pre-upload complete.
Uploading example_file:  |██████████████████████████████ 100% 00:00
Upload Time: 12.94s for 1 files
All uploading jobs have finished.

(8) Upload file to a Shared Folder in the Green Room with tags and attributes

To upload a file with attributes, user must first export an attribute template as a JSON file and fill the template with the desired values.
When uploading files, use the -a/--attribute option to identify the attribute template JSON file.
The Pilot Command Line Interface is capable of validating attribute template. Whenever a template is invalid, including missing non-optional fields, non-existing choice in multiple choice attribute, exceeding length text field and so on, pilotcli will raise error to user and user must update the attribute template and upload again.
Optionally, use the -t/--tag option to include tags. Tags must be listed in a JSON file. Maximum 10 tags per file.
Only files can be uploaded with attributes / tags. Attempting to upload a Folder with -t/--tag or -a/--attribute will result in the following error message:
”Tagging and manifest attaching are not support for folder type.”
Command structure: pilotcli file upload <project name>/<folder type> (users or shared)/<new folder name> <file input location> -t <tag json> -a <attribute json>
- example: pilotcli file upload indoctestproject/shared/IMG ~/Documents/test.csv -t tags.json -a IMG_CoreMetadata.json

$ pilotcli file upload indoctestproject/shared/IMG ~/Documents/test.csv -t tags.json -a IMG_CoreMetadata.json
File attribute validation passed: True
Starting upload of: ./sample_folder/sample_file6
Pre-upload complete.
Uploading sample_file6:  |██████████████████████████████ 100% 00:00
Upload Time: 14.96s for 1 files                                                                   
Attribute attached
All uploading jobs have finished.

(9) Upload file to a Shared Folder in the Core and build lineage with an existing file

Users who have access to the project Core zone can upload a file to a project Core folder and build lineage with an existing file.
Please recall that this command can only be run within a virtual machine that is deployed in the project Core. If you are running the pilotcli tool from your local machine, or from a virtual machine deployed in the greenroom zone, you will run into permission issues. For more details on project zone restrictions, please see: https://indocpilot.atlassian.net/servicedesk/customer/portal/1/topic/bc65e100-23e5-4b50-9a93-3ccb5db1049f/article/1082927 .
To build lineage, user needs to upload a file to the Core zone, then identify the source file with -s/--source-file option. After that, the file will be uploaded to the project core folder as Downstream and build a lineage with identified source file as Upstream. Please note that when listing the source file path, the project code should not be included. See example below for more details.
Command structure: pilotcli file upload <project name>/<folder type> (users or shared)/<folder name> <file input location> -z <zone> (i.e., core or greenroom) -s <source folder type>(users or shared)/<source folder name>/<source file name>
- example: pilotcli file upload indoctestproject/shared/IMG ~/Documents/test.csv -z core
  -s users/blasalandra/test.csv

indocdemo@indoctestproject-demo:~$ pilotcli file upload indoctestproject/users/swangadmin/testfolder0501 ./participants.tsv -s users/swangadmin/dataset_description.json -z core
You are about to transfer data directly to the PILOT Core! In accordance with the PILOT Terms of Use, please confirm that you have made your best 
efforts to pseudonymize or anonymize the data and that you have the legal authority to transfer and make this data available for dissemination and use 
within the PILOT .If you need to process the data to remove sensitive identifiers, please cancel this transfer and upload the data to the Green Room to 
perform these actions. 
To cancel this transfer, enter [n/No] 
To confirm and proceed with the data transfer, enter [y/Yes] 
 [y/N]: y
The manifest file of folder ./resumable_upload_log.json already exist. Do you want to overwrite the existing manifest file? [y/N]: y
Target folder does not exist. Would you like to create a new folder? [y/N]: y
Starting upload of: ./participants.tsv
Pre-upload complete.
Uploading participants.tsv:  |██████████████████████████████ 100% 00:00
Upload Time: 2.66s for 1 files
All uploading jobs have finished.

(10) Upload file to a Shared Folder in the Green Room using threads

Using threads allows for the concurrent uploading of multiple files at the same time.
Specify -td <thread_number> to indicate the number of threads. By default, the number of threads is set to 1.
The max number of thread depends on your machine. The recommend thread number is 3.
There will no UI difference when using threads.
Command structure: pilotcli file upload <project name>/<folder type> (users or shared)/<folder name> <file input location> -td <thread count>
- example: pilotcli file upload indoctestproject/shared/IMG ~/Documents/test.csv -td 3

$ pilotcli file upload indoctestproject/shared/IMG ~/Documents/test.csv -td 3
Target folder does not exist. Would you like to create a new folder? [y/N]: y
Starting upload of: ~/Documents/test.csv
Pre-upload complete.
Uploading ~/Documents/test.csv:  |██████████████████████████████ 100% 00:00
Upload Time: 14.93s for 1 files
All uploading jobs have finished.

(11) Upload file to a Shared Folder in the Green Room using a custom name for the resumable upload file JSON

Upload has an option --output-path ,-o to indicate the name of the resumable upload file JSON. All uploads generate a resumable upload file (resumable_upload_log.json) to use when resuming a stopped upload.
Command structure: pilotcli file upload <project name>/<folder type> (users or shared)/<folder name> <file input location> --output-path <manifest file name and path>
- example: pilotcli file upload indoctestproject/shared/IMG ~/Documents/test.csv
  --output-path ~/Documents/newmanifestfile.json

$ pilotcli file upload indoctestproject/shared/IMG ~/Documents/test.csv --output-path ~/Documents/newmanifestfile.json
Starting upload of: ~/Documents/test.csv
Pre-upload complete.
Uploading ~/Documents/test.csv:  |██████████████████████████████ 100% 00:00
Upload Time: 14.93s for 1 files
All uploading jobs have finished.

(12) Upload folder to a Shared Folder in the Green Room and merge folder contents

If the folder exists in the destination repository, the contents will be checked for duplicated files (case insensitive). Any new files can be uploaded by indicating that you’d like to skip duplicates and continue uploading.
Command structure: pilotcli file upload <project name>/<folder type> (users or shared)/<folder name>/<sub folder name> <folder input location>
- example: pilotcli file upload indoctestproject/shared/IMG/test_folder ~/Documents/test_folder

$ pilotcli file upload indoctestproject/shared/IMG/test_folder ~/Documents/test_folder
Starting upload of: ~/Documents/test_folder
Checking for file duplication...
File duplication check complete.

Some of the selected files cannot be uploaded.

The following files already exist in the upload destination:
test_file_01
Do you want to cancel the upload [N] or skip duplicates and continue uploading [y]? [y/N]: y
Pre-upload complete.
Uploading test_file_02:  |██████████████████████████████ 100% 00:00
Upload Time: 29.86s for 1 files
All uploading jobs have finished.

Command: trash

Options: -z --zone, --permanent
Usage: Send files/folders to trash, permanently delete files from trash, and permanently delete files from Green Room/Core.
Prerequisite:
- User login is required, otherwise error is returned: "The current login session is invalid. Please login to continue."
- Project must be accessible to the user, otherwise, error is returned: "Project Code not found in list. Please verify and try again."
- Project path must include the project folder type (i.e., shared or users).
- User must have permission to send files/folders from relevant top-level folder to trash.

$ pilotcli file trash--help
Usage: pilotcli file trash [OPTIONS] [PATHS]...
  Move files/folders to trash bin.

Options:
  -z, --zone TEXT  Target Zone (i.e., core/greenroom).  [required]
  --permanent      Permanent delete files/folders directly.
  --help           Show this message and exit.

Examples

(1) Move a file/folder from Green Room to Trash

Move a file/folder from a Greenroom folder to the Trash Bin.
Command structure: pilotcli file trash <project code>/<origin folder type> (users or shared)/<origin folder name>/<file or folder name> -z <zone> (i.e., greenroom or core)
- example: pilotcli file trash indoctestproject/shared/IMG/test.csv -z greenroom

$ pilotcli file trash indoctestproject/users/testrun/testfile.txt -z greenroom
Items: ['indoctestproject/users/testrun3/testfile.txt'] have been trashed successfully.

(2) Move a file/folder from Core to Trash

Move a file/folder from a Core folder to the Trash Bin.
Command structure: pilotcli file trash <project code>/<origin folder type> (users or shared)/<origin folder name>/<file or folder name> -z <zone> (i.e., greenroom or core)
- example: pilotcli file trash indoctestproject/shared/IMG/test.csv -z core

$ pilotcli file trash indoctestproject/users/testrun/testfile.txt -z core
Items: ['indoctestproject/users/testrun3/testfile.txt'] have been trashed success

(3) Permanently delete a Green Room file/folder from Trash Bin

Delete Green Room files/folders from the Trash Bin.
Non-administrator users with access to the Trash Bin can only delete the files they have moved to trash. Platform and Project Administrators can delete the files in the Trash Bin from all Project members.
Files/folders in the Trash Bin are deleted based on the zone of their original location (i.e. if the file/folder was moved to trash from the greenroom or core).
Command structure: pilotcli file trash <project code>/trash/<file or folder name> -z <zone> (i.e., greenroom or core) --permanent
- example: pilotcli file trash indoctestproject/trash/test.csv -z greenroom --permanent

$ pilotcli file trash indoctestproject/trash/test.csv -z greenroom --permanent
Items: ['indoctestproject/users/ttrash/test.csv'] have been permanently deleted successfully.

(4) Permanently delete a Core file/folder from Trash Bin

Delete Core files/folders from the Trash Bin.
Non-administrator users with access to the Trash Bin can only delete the files they have moved to trash. Platform and Project Administrators can delete the files in the Trash Bin from all Project members.
Files/folders in the Trash Bin are deleted based on the zone of their original location (i.e. if the file/folder was sent to trash from the greenroom or core).
Command structure: pilotcli file trash <project code>/trash/<file or folder name> -z <zone> (i.e., greenroom or core) --permanent
- example: pilotcli file trash indoctestproject/trash/test.csv -z core --permanent

$ pilotcli file trash indoctestproject/trash/test.csv -z core --permanent
Items: ['indoctestproject/users/ttrash/test.csv'] have been permanently deleted s

(5) Permanently delete a file/folder from Green Room

Delete files/folders from the Green Room.
Command structure: pilotcli file trash <project name>/<folder type> (users or shared)/<folder name>-z <zone> (i.e., greenroom or core) --permanent
- example: pilotcli file trash indoctestproject/users/testrun3/testfile.txt -z greenroom --permanent

$ pilotcli file trash indoctestproject/users/testrun3/testfile.txt -z greenroom --permanent
Items: ['indoctestproject/users/testrun3/testfile.txt'] have been permanently delete

(6) Permanently delete a file/folder from Core

Delete files/folders from the Core.
Command structure: pilotcli file trash <project name>/<folder type> (users or shared)/<folder name>-z <zone> (i.e., greenroom or core) --permanent
- example: pilotcli file trash indoctestproject/users/testrun3/testfile.txt -z core --permanent

$ pilotcli file trash indoctestproject/users/testrun3/testfile.txt -z core--permanent
Items: ['indoctestproject/users/testrun3/testfile.txt'] have been permanently delete

\uD83D\uDCCB Related articles

Page:

Deploying JupyterHub for a Project
Page:

Zone Restrictions with pilotcli
Page:

User Commands - pilotcli user
Page:

Working with Project Files in the Command Line Interface
Page:

Datasets Commands - pilotcli dataset

File Commands - pilotcli file

File Commands: pilotcli file

Help

Command: attribute-export

Examples

(1) Export attributes from a given Project by attribute name:

(2) Export attributes from a given Project by attribute name with options:

Command: attribute-list

Examples

(1) List attributes from a given Project:

(2) List attributes from a given Project with options:

Command: download

Examples

(1) Download file/folder from Green Room

(2) Download file/folder from Core

(3) Download multiple files/folders as a compressed zip

(4) Download file/folder by geid

Command: list

Examples

(1) List files/folders in a Green Room user folder

(2) List files/folders in a Core shared folder

(3) List files/folders in Trash Bin from the Green Room

(3) List files/folders in Trash Bin from the Core

Command: metadata

(1) Download a file with general metadata, attributes, and tags.

Command: move

Examples

(1) Move a file from one folder to another within a Green Room Project

(2) Move a file from one folder to another within a Core Project

(3) Rename a file in a Green Room folder

Command: resume

Examples

(1) Resume the upload with resumable upload file (resumable_upload_log.json)

Command: upload

Examples

(1) Upload file to a Shared Folder in the Green Room (on Windows)

(2) Upload file to a Shared Folder in the Green Room (on Linux/Mac OS)

(3) Upload file to a Shared Folder in the Core

(4) Upload folder to a Shared Folder in the Green Room

(5) Upload file to an existing folder in a Shared Folder in the Green Room

(6) Upload folder as zip to a Shared Folder in the Green Room

(7) Upload folder to a Shared Folder in the Green Room and create a new folder

(8) Upload file to a Shared Folder in the Green Room with tags and attributes

(9) Upload file to a Shared Folder in the Core and build lineage with an existing file

(10) Upload file to a Shared Folder in the Green Room using threads

(11) Upload file to a Shared Folder in the Green Room using a custom name for the resumable upload file JSON

(12) Upload folder to a Shared Folder in the Green Room and merge folder contents

Command: trash

Examples

(1) Move a file/folder from Green Room to Trash

(2) Move a file/folder from Core to Trash

(3) Permanently delete a Green Room file/folder from Trash Bin

(4) Permanently delete a Core file/folder from Trash Bin

(5) Permanently delete a file/folder from Green Room

(6) Permanently delete a file/folder from Core

\uD83D\uDCCB Related articles

(4) Download file/folder by `geid`

(1) Resume the upload with resumable upload file (`resumable_upload_log.json)`