Storage¶
Storage Overview¶
There are three complementary filesystems where you can store your data.
| Name | Path | Alias | Backup | Purged | Usage | Disks |
|---|---|---|---|---|---|---|
| home | /home/$USER | ~ | No | No* | 400 GB (limit) | Redundant, SSD |
| scratch | /scratch/$USER | ~/scratch | No | 30 days | 20 TB (limit) | Redundant, HDD |
| scalable storage | /shares/<PROJECT> | see below | No | No | with cost contribution (quota) to increase quota, contact us | Redundant, HDD |
*Once your UZH Active Directory (AD) account is deleted, or your user is removed from all project associations, your /home directory will be scheduled for deletion after 30 days.
To keep your data on the ScienceCluster, move it to your /shares project folder before your AD account is deactivated.
Filesystems¶
Tip
To see an overview of your storage usage, run the command quota from a login node.
Warning
If you exceed the quota, you will not be able to write to that file system. Check the FAQs for support on how to clean up your file storage if you are over-quota.
Danger
There are absolutely no filesystem-wide backups or archives. Any accidental file deletions or modifications are irreversible. We strongly recommend users back up and/or archive important data on a regular basis.
Home¶
Each user has a home directory (~/ or /home/$USER) where configuration files, source code, software environments, and other important files can be stored. For reference, home directories are stored on solid state drives (SSD) and are therefore suggested for folders with many files.
Scratch¶
The scratch filesystem (~/scratch or /scratch/$USER) is for the temporary storage of large input data files used during your calculations. Each user has a quota of 20 TB. The maximum file size is limited to 10 TB. Please note that this filesystem is meant for temporary storage only. According to the service agreement, any files that have not been accessed within 30 days will be deleted.
Warning
Any attempts to circumvent the automatic deletion policy may be considered abuse of the system.
Scalable Storage¶
Scalable group storage has a cost contribution that is based on the actual usage. The default permissions are set so that each member of the project has access to the shared folder, which can be found at this path: /shares/<PROJECT>. (In this case, replace <PROJECT> with your actual project name.)
You can create a symlink called shares in your home directory that points to this shared group folder:
Further Storage Options¶
UZH Central IT (ZI) offers further storage options. For more details, please check General Topics → Data Storage.
- Data Archiving
- Data Publishing
- Network Storage (SMB- and NFS-based) with high availability and backup
- For connection instructions to NAS share, please check article How to Backup your ScienceCluster data to a NAS.
FAQs¶
I am over-quota. How can I clean up my file storage?¶
Consider storing large files in your scalable storage folder, which is in your project space and can be found by running the command quota.
Folders that typically grow in size with cache or temporary files are ~/.local and ~/.cache. To find the storage used in all subfolders in your /home/$USER folders, run:
If you cannot login anymore into the cluster you can still connect using terminal only:
and then remove files or kill processes if needed.Conda¶
To clean up cached installation packages from Conda environments, see here.
Apptainer¶
Check Apptainer cache set up and clean up instructions in this article.
Framework folders¶
Certain software frameworks (e.g., HuggingFace) cache files programmatically, which can be cleaned with their own commands. For example, with HuggingFace consider using:
You can find corresponding information on such commands in the framework specific documentation, such as this page for information on HuggingFace cache cleaning.