Network shares and slow storage

A network share is slower than a local disk, can be unplugged without warning and is often somebody else’s. This guide says where to put the project, how Ingence treats links, and what keeps working when the share is gone.

Put the project on your own disk

A project is written to on every run, so a local disk is faster and is there when the share is not. Keep the project out of the folder it reads. See Privacy and your data for why.

Look before you commit

ingence index -p PROJECT --dry-run --seconds 60

This walks a share’s metadata for a minute, reports how fast it goes and what it holds, and writes nothing. If the answer is “slowly,” you will know before you wait for a whole census.

A census can be stopped

ingence index can be interrupted at any moment. A stopped census publishes nothing and leaves the previous result current, so you can stop it on a slow share without leaving the project in a bad state.

Links

Ingence records a link and never walks into it. By default, the census asks whether each link’s target is there, so that a link whose target is gone can be told apart from one whose target is elsewhere. It follows a chain one link at a time, and only inside the folder being read. The first link whose text points outside it is read from its text and never followed, whatever the setting.

On a slow share, each of those questions waits on the share. To turn them off, put this in the project’s settings.yaml:

scan:
  resolve_symlinks: false

Each link is then recorded as a link, with its target unasked. The next census uses the new setting.

Mounted drives inside the folder

The walk stays on one disk by default (scan.one_file_system). It does not cross into another drive or share that is mounted inside the folder.

What a dive reads on a share

A hash is a fingerprint made by reading a file’s bytes. Ingence uses hashes to prove that two files are identical. Reading bytes across a network is slow, and it loads the share for everyone else who uses it. These are the choices, from least to most read.

• init --depth metadata opens nothing and hashes nothing. The census asks the share only for names, sizes and dates. This is the choice for a share that is not yours (Read someone else’s folder).

• The default (standard depth, hash mode need) opens common formats within bounds, and it hashes only what comparing files needs. It reads a small sample of every file whose size another file shares, and the whole of every file whose sample another file’s sample matches, so that it can prove two files identical. A file whose size nothing else has is not read for this. This happens on a share too. The rule that holds back hashing on a share applies to the modes that hash every file, and never to need, because comparing and finding duplicates is what a dive on a share is for. Hashing every file happens only if you choose a mode that does it: hash.mode set to quick, full or adaptive, or hash.need_from set to depth. Nothing does it by default.

• deep reads more content. It reads broadly, takes larger samples of each file and has no time budget. Its hashing is the same as the default’s unless you chose one of those modes. If you did, a dive at standard leaves a share’s files unhashed and a dive at deep hashes them. Choose deep deliberately, and not on a share that other people depend on.

One setting in the project’s settings.yaml, hash.skip_remote_filesystems, changes this. What you set there is your choice and stands at every depth. To withhold every digest on a share, need’s as well:

hash:
  skip_remote_filesystems: always

This stops all hashing on a share. A dive still opens files there to read headers, text and tables, so use init --depth metadata when no file should be opened at all.

Setting it to false hashes shares at every depth. Under the default need mode that changes nothing, because need already hashes on a share. It matters only if you chose a mode that hashes every file, where it makes standard hash a share as deep does. Left unset, the depth decides, as above.

When the share is away

A census and a dive need the folder and refuse a folder they cannot reach, before writing anything. Re-reading names (ingence index --rebind), ingence infer, answering questions and applying the answers need only the project, so they work with the drive unplugged or the share unmounted. So do the Dashboard’s pages and its Re-run. The Dashboard touches the folder itself in two cases only. One is the first run it starts from its landing page, which lists the folder you name and then reads it. The other is the check behind the Graph page’s reorganization planner, which asks the share whether the files in a plan are still there, their sizes and dates, whether a folder can be written to and how much room is free. It opens no file, and it answers “unknown” when the share is away. The last finished result stays readable, and an interrupted command leaves the previous result current.

Cloud-synced folders

A folder kept in step with OneDrive, Dropbox, iCloud or a similar service may hold files that exist only in the cloud. Opening such a file can make the sync client download it. The census opens no file. A dive does. Use init --depth metadata until you have checked what your client does.