Settings#

This page is generated from the code and must not be edited by hand.

A project’s settings.yaml holds only the values a person chose. Every setting not named there takes the default shown here. A dotted name is a nested key: hash.mode is the key mode inside the key hash.

There are 168 settings in 15 groups. The Dashboard shows the same list on its Engine tab, with the same words. Copy puts the setting, at its default, on your clipboard as the lines of settings.yaml that hold it: change the value and paste it into the file. If the file already has the group (for example scan:), put the copied line under that group and leave out the group’s name from the copy. A file that names one group twice is read without an error, and only the last of them counts.

How deep a run reads#

Setting

Default

May be

What it is for

depth

standard

metadata: never open a file; standard: open common formats, within bounds; deep: read broadly, small files hashed whole

How far a dive goes unless told: never open files, standard, or deep

Walking the folders#

Setting

Default

May be

What it is for

scan.exclude_globs

[".git", ".svn", ".hg", "__pycache__", "node_modules", ".Trash", ".Trashes", "$RECYCLE.BIN", "System Volume Information", ".ingence-case-probe"]

a list of words

Folder and file name patterns the walk skips, with everything beneath them

scan.flush_every_records

4096

whole number, at least 1

How many records gather before they are written to disk

scan.flush_every_seconds

30.0

number, more than 0

Longest time records wait before being written to disk

scan.follow_moves

on

on or off

Whether a file that moved is recognized, keeping its readings and decisions

scan.follow_symlinks

off

fixed; it cannot be changed

Whether shortcuts to other places are followed; always off, cannot be changed

scan.include_hidden

off

on or off

Whether hidden files and folders, whose names start with a dot, are walked

scan.index_recognised_outputs

off

on or off

Whether earlier Ingence projects and runs found inside the data are indexed as data

scan.keep_dive_readings

on

on or off

Whether an unchanged file keeps what an earlier dive read of it

scan.max_depth

not set

whole number, or left empty

How many folders deep the walk goes; blank means no limit

scan.move_candidates_cap

64

whole number, at least 1

Past this many files alike in name, size and time, look-alikes are not paired for moves

scan.move_folder_min_files

2

whole number, at least 1

Fewest of a folder’s files that must arrive together to count the folder as moved

scan.move_folder_min_share

0.5

number, more than 0, at most 1

Share of a folder’s contents that must match one to one to count it moved

scan.moves_by_file_number

on

on or off

Whether, on one disk, the disk’s own number for each file decides move or copy

scan.moves_confirmed_by_dive

on

on or off

Whether a dive checks a carried move by comparing the file’s hash

scan.moves_refuse_copies

off

on or off

Whether a copy, its original removed, starts afresh instead of taking the original’s readings and answers

scan.one_file_system

on

on or off

Whether the walk stays on one disk, never crossing into other mounted drives

scan.resolve_symlinks

on

on or off

Whether a shortcut’s target is checked to see if it still exists

scan.workers

8

whole number, from 1 to 256

How many folders the walk lists at once

Comparing files#

Setting

Default

May be

What it is for

hash.algorithm

blake3

text

The method used to hash file contents

hash.chunk_bytes

1048576

whole number, at least 4096

How much of a file is read at a time while hashing

hash.exclude_extensions

empty list

a list of words

File extensions never hashed

hash.full_hash_max_bytes

2147483648

whole number, at least 0

Largest file hashed whole; bigger ones are only sampled, so never proven identical

hash.include_extensions

empty list

a list of words

When filled, only these file extensions are hashed

hash.max_file_bytes

not set

whole number, or left empty

Files larger than this are never hashed; blank means no limit

hash.mode

need

none: nothing; quick: a sample of every file; full: every file whole; adaptive: whole files below a size, a sample above; need: only what comparing needs

What a dive hashes: only what comparing files needs, every file, or none

hash.need_from

comparison

comparison: what comparing files needs; depth: what the dive’s depth names

What ‘need’ goes by: what comparing files needs, or the dive’s depth

hash.skip_remote_filesystems

not set

none: left to the depth; True; False: hash shares too; always: never hash on a share

Hashing across network shares: left to the depth, always done, or never done

Reading inside files#

Setting

Default

May be

What it is for

extraction.adapters

empty list

a list of words

Which file readers may run; empty allows every available one

extraction.beneath_programs

read_last

read_last: after everything else; not_read: never; read_in_turn: in turn, like any file

When files inside installed programs are read: after everything else, never, or in turn

extraction.byte_scan_sections

empty list

a list of words

Dictionary sections whose words are searched for inside file contents; empty means none

extraction.clock_only_extensions

["nd2"]

a list of words

File extensions read only for their timestamp when the ND2 clock-only option is on

extraction.deepening_budget_seconds

not set

number, or left empty

Total time a dive may spend reading inside files; each dive’s depth sets it

extraction.enabled

off

on or off

Whether a dive reads inside files at all; each dive’s depth sets it

extraction.exclude_extensions

empty list

a list of words

File extensions whose contents are never read

extraction.freshness_checks_identity

on

on or off

Whether a reading is also discarded if the file was replaced while being read

extraction.include_extensions

empty list

a list of words

When filled, only these file extensions are read inside

extraction.max_expansion_depth

4

whole number, at least 0

How many levels of archives within archives may be opened

extraction.max_expansion_members

10000

whole number, at least 0

Most files one archive may yield when opened

extraction.max_expansion_ratio

100.0

number, more than 0

How many times its own size an archive’s contents may grow to when opened

extraction.max_file_bytes

not set

whole number, or left empty

Files larger than this are not read inside; blank means no limit

extraction.max_sample_bytes

4194304

whole number, at least 0

How much of one file a reading may take in at most; each dive’s depth sets it

extraction.max_text_chars

200000

whole number, at least 0

Most characters of text kept from one file

extraction.metadata_first

on

on or off

Whether a large file’s header is read first, and its body only if still needed

extraction.nd2_clock_only

off

on or off

Whether ND2 files are read only as far as their header and timestamp

extraction.promote_container_members

on

on or off

Whether files found inside archives become items of their own

extraction.spend_order

family_round_robin

smallest_first: smallest first; family_round_robin: file types in turn; extension_round_robin: extensions in turn

The order a dive reads files: file types in turn, extensions in turn, or smallest first

extraction.subprocess_memory_bytes

1073741824

whole number, or left empty

Memory ceiling for a reading run apart from the engine; blank means none

extraction.subprocess_timeout_per_gigabyte_seconds

120.0

number, at least 0

Extra seconds of reading time allowed per gigabyte of file

extraction.subprocess_timeout_seconds

60.0

number, more than 0

Seconds one file’s reading may run before it is stopped, plus the per-gigabyte allowance

extraction.supervise_in_process_above_bytes

262144

whole number, at least 0, or left empty

Above this size, every reading runs apart so its time limit applies; blank means never

extraction.text_sketch_bytes

262144

whole number, at least 1024

How much of a text file is read to spot near-duplicates

extraction.use_subprocess_isolation

on

on or off

Whether readers that ask for it run apart, so one bad file cannot hang the run

Reading names#

Setting

Default

May be

What it is for

dictionary.bind_terms

on

on or off

Whether the Dictionary is asked what each part of a name means

dictionary.burst_gap_factor

4.0

number, more than 1

How many times longer than nearby pauses a pause must be to end a burst

dictionary.burst_gap_window

9

whole number, at least 3

How many neighboring pauses the typical pause is judged against

dictionary.case_reading.enabled

on

on or off

Whether spellings joined at a capital are read by evidence, rather than always as pieces

dictionary.case_reading.unknown_whole

pieces

whole: one unknown word, asked about; pieces: its known pieces

For a one-word spelling the Dictionary lacks: treat it as unknown, or read its known pieces

dictionary.case_reading.whole_over_alone

0.5

number, more than 0

How often a joined spelling must appear, against its rarest piece alone, to be one word

dictionary.characterise_organisation

on

on or off

Whether each folder’s name patterns, time span and bursts of activity are described

dictionary.date_series

settle

settle: sibling dates settle it; off: a dated folder alone

Whether sibling short dates together settle an ambiguous day, month and year order

dictionary.domain_packs

all

all: every domain pack; named: only the packs named

Whether every shipped domain pack loads (but those refused), or only the ones named

dictionary.domain_packs_named

empty list

a list of words

The shipped domain packs to load when only named ones are chosen

dictionary.files_to_accept_a_pack

3

whole number, at least 0

How many files in an instrument’s format (ND2 for microscopy) accept its pack without asking, 0 never. Once a pack is accepted, the packs the project chose (accepted ones and the laboratory’s own) stand above the core and every other shipped pack level with it: a word they spell differently is asked once

dictionary.glued_runs

letter_led

letter_led: one word if it starts with a letter; whole: always one word; split: letters and numbers apart

How letters and digits written together are read: by what leads, as one word, or split

dictionary.glued_words.half_minimum

4

whole number, at least 1

The fewest letters each part may have when a run of two joined known words is divided

dictionary.glued_words.reading

divide

divide: divided into its two words; whole: one word

Whether a run of two known words written together, like channelhoechst, is divided

dictionary.key_value_pairs

2

whole number, at least 1

How many pairs like sub-01 a name needs before each reads as a numbered label

dictionary.packs

empty list

a list of words

Vocabulary packs the lab installed that a run loads, in order

dictionary.plausible_years

[1990, 2035]

a list of numbers

Older year range for dates in names; kept so old projects load, no longer used

dictionary.plural_minimum

4

whole number, at least 2

The fewest letters a word needs before it is read as the plural of a known word

dictionary.record_name_shapes

on

on or off

Whether where each part sat within a name is recorded Read by the rules replicates_of_one_condition, same_name_identity, same_name_identity_within_experiment.

dictionary.six_digit_dates

year_first

year_first: a date, year first; none: a number

Whether six digits alone are read as a year-month-day date or as a number

dictionary.spelling.candidates

3

whole number, at least 1

How many known words are offered beside a word that may be a misspelling

dictionary.spelling.edits

2

whole number, from 1 to 3

How many typing slips apart a word may be from a known word it is offered as

dictionary.spelling.minimum_letters

6

whole number, at least 3

The fewest letters a word needs before a known word is offered as its spelling

dictionary.tokenise_names

on

on or off

Whether every name is split into words and the result recorded

dictionary.years

[2013, "present"]

a list of years or the word present

The years a date in a name may fall in, first to last or the present

The run’s journal#

Setting

Default

May be

What it is for

journal.enabled

on

on or off

Whether a run keeps a detailed journal of its settings, timings and decisions

journal.per_entry

on

on or off

Whether the journal records one line for each file a dive reads

journal.timings

on

on or off

Whether the journal records how long each step took for each file

Archive listings#

Setting

Default

May be

What it is for

listing.declared_links

withheld

withheld: held back, for scoring; recorded: entered in the Index

Whether links an archive’s listing declares enter the Index, or are held back for scoring

listing.pause_seconds

0.5

number, from 0 to 60

Seconds to wait between two requests to one archive

listing.read_archive_clocks

off

on or off

Whether to ask an archive for each file’s date where its listing gives none

listing.user_agent

Ingence listing ingest (research pilot; local, read-only)

text

What each request to an archive says it comes from

Background jobs#

Setting

Default

May be

What it is for

jobs.executor

subprocess

subprocess: a program of its own; thread: inside the Dashboard

Whether background jobs run as their own program or inside the Dashboard

jobs.heartbeat_seconds

2.0

number, more than 0, at most 60

How often a running job says it is still working

jobs.logs_kept

100

whole number, from 1 to 100000

How many jobs’ logs are kept before older ones are removed

jobs.retry_limit

2

whole number, from 0 to 20

How many times an interrupted job is tried again before it is marked failed

jobs.silent_seconds

60.0

number, more than 0, at most 3600

How long a job may go unheard before it is reported silent

jobs.stop_silent

on

on or off

Whether a job silent too long is stopped and tried again

jobs.stop_silent_seconds

900.0

number, more than 0, at most 86400

How long a job must be silent before it is stopped

Applying the rules#

Setting

Default

May be

What it is for

infer.entries_per_chunk

16384

whole number, at least 1

How many items are held at once when rules are applied folder by folder

infer.memory_margin_fraction

0.25

number, at least 0, less than 1

Share of the machine’s memory left free when deciding whether to go folder by folder

infer.projected_bytes_per_entry

17408

whole number, at least 1

Estimated memory one item needs, used to decide whether to go folder by folder

infer.streaming

never

auto: folder by folder when memory needs it; always: always folder by folder; never: all at once

Whether rules are applied folder by folder to save memory: when needed, always, or never

Completeness and no analysis found#

Setting

Default

May be

What it is for

completeness.forgotten_at_least

0.5

number, more than 0, at most 1

Reason ‘forgotten’: the share of the rest of its theme’s raw data found analyzed, at least

completeness.judged_from_acquisitions

5

whole number, from 1 to 1000000

How many raw acquisitions an experiment needs found before it can be listed as having no analysis found

completeness.listed_below_share

0.5

number, more than 0, at most 1

Below this share of its raw acquisitions found followed by analysis, an experiment is listed as having no analysis found

completeness.own_analysis_files

5

whole number, from 1 to 1000000

How many analysis files found count as an analysis of the experiment’s own

completeness.own_result_tables

3

whole number, from 1 to 1000000

How many results tables found count as an analysis of the experiment’s own

completeness.stopped_below

0.2

number, at least 0, less than 1

Reason ‘a line that stopped’: the share of the rest of its theme’s raw data found analyzed, under

The design grid of an experiment#

Setting

Default

May be

What it is for

design.cells_listed_at_most

2000

whole number, from 1 to 1000000

The most combinations of an experiment’s grid listed as not found; a larger grid is counted and not listed

design.enabled

off

on or off

Off by default while it is measured; turn it on to see a grid of the conditions an experiment’s names vary

design.factor_levels_at_most

50

whole number, from 2 to 1000000

The most different values a part of a name may take and still be drawn as a condition; a part with more is taken for an identifier or a frame number

design.include_unclassed

off

on or off

Whether a file that no group of repeats holds, such as the only file of its condition, is drawn in the grid when its name is laid out like the others; on, scripts and documents with alike names make grids too

design.merge_exception_share

0.1

number, from 0 to 0.5

The share of an experiment’s files that may break the pairing of two parts of a name that go together, such as GFP and 488

design.merge_minimum_observations

2

whole number, from 1 to 1000000

How many files each value of a part needs before the part may be joined with another that always goes with it

design.merge_parts

on

on or off

Whether two parts of a name that always go together, such as GFP and 488, are drawn as one factor

Themes of experiments#

Setting

Default

May be

What it is for

themes.broad_resolutions

[0.05, 0.1, 0.15, 0.2, 0.3, 0.4, 0.5, 0.7, 1.0]

a list of numbers

Coarseness levels tried when grouping themes into broad themes

themes.chance_null

nearest_ten

nearest_ten: each shuffled experiment’s ten nearest; matched: as many as the graph joins; the single nearest for placement

Which similarities in the shuffled copies the chance cutoffs are read from

themes.chance_quantile

0.75

number, more than 0, less than 1, or left empty

How much closer than chance an experiment must be to the rest of its theme to be placed; blank places all

themes.chance_replicates

1

whole number, from 1 to 100

How many shuffled copies of the experiments the chance cutoffs are drawn from

themes.collection_level

0

whole number, from 0 to 8

What counts as one collection when weighing them: each root, or each folder just below a root

themes.collection_weight_floor

10

whole number, at least 1

Fewest experiments a collection is weighed as if it had, so a tiny collection does not outweigh the rest

themes.collection_weight_power

0.5

number, from 0 to 1

How much a large collection outweighs a small one: 1 counts every experiment alike, 0 every collection alike

themes.direction_ratio

1.5

number, more than 1

How much a theme’s recent share must grow, or shrink, to be called rising or falling

themes.direction_recent_years

2

whole number, at least 1

How many latest years a theme’s recent share is taken over

themes.edge_chance_quantile

0.75

number, more than 0, less than 1, or left empty

How much closer than chance two experiments of different collections must be to be joined; blank joins every nearest neighbor

themes.edge_chance_quantile_within

not set

number, more than 0, less than 1, or left empty

How much closer than chance two experiments of one collection must be to be joined; blank joins every nearest neighbor

themes.final_placement

medoid

medoid: the seed most like the others; first: seed 0

Which of the drawn seeds is shown: the one most like the others, or the first

themes.fine_resolutions

[0.3, 0.5, 0.7, 1.0, 1.4, 2.0, 3.0]

a list of numbers

Coarseness levels tried when drawing themes; the steadiest readable one is kept

themes.fold_unknown_plurals

on

on or off

Whether a word the Dictionary does not know and its plural count as one word when both appear

themes.how_kinds

["channel", "connective", "count", "dataset", "date", "experiment", "file_role", "format", "identifier", "measurement", "part", "process", "project", "quality", "quantifier", "replicate", "stage", "statistic", "status", "unit", "version"]

a list of words

Kinds of Dictionary word that say how a file was made, left out of themes

themes.ignored_words

empty list

a list of words

Words left out of every theme

themes.label_bootstrap

100

whole number, from 0 to 1000

How many times the experiments are drawn again to see how steady each label is; 0 draws none

themes.label_method

tfidf

tfidf: TF-IDF; log_odds: log-odds against the rest

How a theme’s words are ranked: by TF-IDF, or by the log-odds of the word inside the theme against the rest

themes.label_prior_strength

100.0

number, more than 0

How strongly the log-odds ranking leans on how common a word is overall

themes.label_sense_specificity

0.5

number, from 0 to 1

Share of a Dictionary word’s use that must fall in a theme for it to lead the theme’s label

themes.label_words

3

whole number, at least 1

How many words a theme’s label shows

themes.lsa_dimensions

not set

whole number, at least 2, or left empty

Most dimensions experiments’ words are summarized in before they are compared; blank grows them with the experiments

themes.max_share

0.5

number, more than 0, at most 1

A word in more than this share of experiments is background, not a theme word

themes.min_experiments_sense

2

whole number, at least 1

Fewest experiments that must share a word the Dictionary knows for it to count

themes.min_experiments_unknown

3

whole number, at least 1

Fewest experiments of one collection that must share an unknown word for it to count

themes.min_theme_documents

2

whole number, at least 1

Fewest experiments with different words a theme needs; experiments that read the same words are not a theme on their own

themes.min_theme_size

3

whole number, at least 1

Fewest experiments a theme needs; smaller groups are listed as not placed

themes.neighbours

not set

whole number, at least 1, or left empty

How many closest experiments each one is joined to; blank chooses from the number of experiments

themes.person_level

1

whole number, from 1 to 8, or left empty

Which folder level below each root is a person’s, for who else; blank for nobody

themes.person_min_experiments

3

whole number, at least 1

Fewest experiments a person needs in a theme, or with a word, to count there

themes.resolution_edge_steps

2

whole number, from 0 to 8

How many coarseness levels past the last one tried are tried when the steadiest readable one is the last

themes.seeds

8

whole number, from 2 to 64

How many times the themes are drawn, to measure how steady each one is

themes.specific_max_share

0.8

number, more than 0, at most 1

Most of a theme’s experiments a word may be in and still be one of its specific words

themes.specific_min_specificity

0.5

number, more than 0, at most 1

Share of the experiments with a word that must be in the theme for it to be the theme’s

themes.subfolder_weight

1.0

number, at least 0

How much a word in a folder inside an experiment counts

themes.sublinear_counts

off

on or off

Whether a word written many times counts less than in proportion

themes.substitute_max_together

0.1

number, from 0 to 1

Most often two words may appear together and still be read as one replacing the other

themes.substitute_min_similarity

0.35

number, from 0 to 1

How alike the company two words keep must be to read one as replacing the other

themes.theme_words

6

whole number, at least 1

How many distinctive words are kept for each theme

themes.title_weight

3.0

number, at least 0

How much a word in an experiment folder’s own name counts

themes.titles_or_acquisitions_only

on

on or off

Whether a word must appear in a folder name or an acquisition’s name to count

themes.vertex_weight_in_graph

off

on or off

Whether identical experiments, gathered into one, keep their number’s weight when themes are drawn

themes.year_min_experiments

2

whole number, at least 1

Fewest experiments a year needs to be described in the year profile

The question sheet#

Setting

Default

May be

What it is for

questions.atypical_floor

2

whole number, from 0 to 10000

A pattern of made files seen in this many pairs or fewer is asked about as unusual

questions.atypical_most

5

whole number, from 0 to 10000

The most questions about unusual pairs on the sheet

questions.atypical_share

0.001

number, from 0 to 1

A pattern seen in this share of all pairs or fewer is unusual, where that is more than the floor

questions.audit_cover

0.95

number, more than 0, at most 1

The share of the rules’ readings the first questions about them are chosen to cover

questions.audit_first

20

whole number, from 0 to 10000

The most questions about the rules’ readings asked first

questions.pattern_cover

0.9

number, more than 0, at most 1

The share of the proposed made-from pairs the first pattern questions are chosen to cover

questions.pattern_first

12

whole number, from 0 to 10000

The most questions about patterns of made files asked first

questions.sheet_budget

50

whole number, from 1 to 10000

The most questions on the question sheet

questions.sheet_tied_words

10

whole number, from 0 to 10000

The most words standing between two meanings the sheet asks about

Tables with the same columns#

Setting

Default

May be

What it is for

columns.files_listed

20

whole number, from 0 to 500

How many files each list on a table’s page names; the count beside the list is always the whole

columns.inclusion

on

on or off

Whether tables are listed whose columns contain another table’s columns

columns.inclusion_minimum_columns

3

whole number, from 1 to 1000

Fewest columns a table needs to be listed as contained in another table

columns.minimum_columns

2

whole number, from 1 to 1000

Fewest columns a table needs to be listed as having the same columns as other tables, used only when the rule for tables with the same columns is not in force