Dealing with Large Files

It is often necessary to deal with files large enough that available space becomes an issue. On Babel, for example, most of you probably have a quota of only a few megabytes. It is therefore useful to know:

  1. where to put large files;
  2. how to avoid making unnecessary copies of files.

The place where you can put large files is the /tmp directory. This is a directory provided for temporary files. All users can read and write to this directory. /tmp is not included in your quota, so you can use as much space there as you like.

However, /tmp is intended for TEMPORARY files. You can't count on files remaining untouched. On some systems /tmp is cleared out periodically, e.g. every Sunday. On other systems, the system manager clears out /tmp irregularly, whenever it becomes cluttered. Therefore, once you have done your work there, you should copy anything that you want to keep, such as your results, to your own directory.

One way to keep your files together and avoid interference with other users is to create your own subdirectory within /tmp. The mkdir command will do this for you. For example, if I have changed directory to /tmp, I could make a subdirectory called "wjposer" by giving the command;

mkdir wjposer

You can often avoid creating extra copies of large files by making a link to them. Suppose, for example, that you want to work on a file called:


from your home directory, without having to type that long filename every time. You can make it look like the file is in your home directory, without copying it, by making a link to it using the following command:

ln -s /pkg/ling/etc/httpd/htdocs/courses/Spring_2003/ling538/Data/SwahiliVerbs SwahiliVerbs

If you list your directory, you will now have an entry SwahiliVerbs, to which you can refer just as if the file were really in your directory. However, you have actually created a symbolic link, that is, a pointer to the actual file. Here is the result of typing "ls -l SwahiliVerbs" in my own directory after creating such a link.

lrwxrwxrwx 1 wjposer penn 72 Feb 23 17:18 SwahiliVerbs -> /pkg/ling/etc/httpd/htdocs/courses/ling538/Data/SwahiliVerbs

Note: in some cases you can achieve the same effect using a "hard" link, which is created by giving the same commmand without the -s flag. That is not possible in this case because the original file and my directory reside on different filesystems. Hard links cannot cross filesystems, while symbolic links can.