Uploading a lot of files?
Your project may have thousands of existing files of diverse types. As you are planning to move your research into the workspace, understanding how the upload process works can help you plan how to do this most effectively. For example, you might consider one of the following approaches:
- We recommend that you upload no more than 500 files at a time (just as you probably wouldn’t between your desktop and any other shared file server).
- If you have a virtual desktop add-on to your workspace, using .zip files to batch uploads into manageable chunks is one way to manage this. You will be able to unpack these later in the workspace, and the upload process will be easier.
- File uploads via the web UI have an upload limit of 1GB.
- Individual files (including .zip files) that are larger than 250GB should not be uploaded into the workspace using the methods described. If you have files that are over 250GB, please get in touch with your Data Steward Team who will be able to help plan the data migration.
See the table below for a summary of different types of source data, guidance on where to store it, and how you can access the data once it has been uploaded into the workspace.
|Source data to be uploaded||File extensions||Typical size per file||Purpose||Workspace folder||Data mapping applied?||Accessed from|
|Web interface||Virtual desktop|
|Tabular data||.csv||1000s of rows and columns < 10MB||Database analysis||Files||Workspace database||Yes||Yes|
|Analysis scripts||.r, .sql||100 – 500kB||Reproducible statistics||Files||Workspace file system||Yes||Yes|
|Text, pdf documents and small images||.txt, .doc, .pdf, .png, .jpg||2MB||Project communication and reports||Files||Workspace file system||Yes||Yes|
|Large image files, image series, genomic data, executable files for tool installation, other non-structured data||.png, .jpg, .vcf, .exe||100MB – 250GB||Raw data for analysis and information extraction||Blobs||Workspace file system||Yes||No|