Skip to content

Quick start

The shortest path from an empty directory to a taxonomically annotated table. Five steps: clone, create the launcher environment, edit the config, preview the plan, run it.

# 1. Clone
git clone https://github.com/iLivius/MetaFlux.git
cd MetaFlux

# 2. Create the launcher environment (Snakemake + conda support)
conda create -n snakemake -c conda-forge -c bioconda snakemake
conda activate snakemake

# 3. Edit config/config.yaml — set `mode`, input/output paths, and references
#    (amplicon: primer FASTAs; shotgun: a Kraken2 database, or sylph databases)

# 4. Dry-run to check the plan, then execute
snakemake --sdm conda --cores 16 --configfile config/config.yaml --dry-run
snakemake --sdm conda --cores 16 --configfile config/config.yaml

Run these from the repository root: Snakemake looks for workflow/Snakefile there, and relative paths in the config resolve against the directory snakemake is invoked from, not against the config file. Absolute paths are safest.

Steps 1 and 2 are covered in more detail under installation, including where the reference databases come from and what --sdm conda builds.

Step 3 in detail — the five settings that define a run

config/config.yaml is long, but most of it is tuning that can stay as shipped. Five settings decide what the run actually is; the box at the top of the template lists the same ones:

Setting What it decides
mode amplicon or shotgun — see choosing a mode
amplicon.type 16S, ITS, 18S, gyrB, rpoB or COI (amplicon runs only)
shotgun.profiler kraken2 (the default), sylph or both (shotgun runs only) — see sylph
input.fastq_dir The directory holding the paired FASTQ files
output.out_dir Where every result is written

Then whatever the chosen mode needs before it can start, and MetaFlux downloads none of it: the two primer FASTA files for amplicon (amplicon.primers.fwd / .rev). For shotgun, the Kraken2 database directory (references.kraken_db) with kraken2 or both, and the sylph databases (references.sylph_databases, with their lineage tables in references.sylph_taxonomy_dir) with sylph or both; a sylph-only run needs no Kraken2 database.

Input files are found by pattern, not listed by hand. MetaFlux globs fastq_dir for {sample}_R1 / {sample}_R2 pairs — shotgun also accepts the SRA-style {sample}_1 / {sample}_2 — and the part before the mate suffix becomes the sample name used in every output table. All files must share one extension, and in practice that extension has to be fastq or fastq.gz. The start-up check also lets fq and fq.gz through, but every rule that reads a raw file resolves its path as .fastq.gz or .fastq, so a directory of .fq.gz files passes validation and then fails later on an input the rules cannot resolve.

Keep the template, edit a copy

The workflow always loads config/config.yaml as a base layer, and a file passed with --configfile is applied on top of it. So copying the template and editing the copy works cleanly:

cp config/config.yaml config/config.local.yaml
snakemake --sdm conda --cores 16 --configfile config/config.local.yaml

.gitignore tracks only config/config.yaml; any other config/*.yaml stays out of git, so a run config with real paths in it is never committed by accident.

Step 4 in detail — why the dry-run matters

--dry-run makes Snakemake work out every job it would run and then stop without running any. It takes seconds. MetaFlux's own checks run while that plan is built, so a dry run stops on the same configuration errors a real run would stop on, and it also lists the jobs before any conda environment is built or reference database downloaded. What is checked before the first job starts:

  • mode is one of amplicon / shotgun.
  • fastq_dir yields at least one {sample}_R1 file (or {sample}_1 in shotgun mode), all with one consistent extension, and sample names without characters Snakemake cannot use as wildcards (a wildcard is the placeholder — here {sample} — that a rule is instantiated over, once per sample).
  • Amplicon: amplicon.type matches a marker; both primer FASTA files exist; taxonomy.method is available for that marker (gyrB and rpoB have no SINTAX reference, so only rdp works for them, and COI has no rdp trainset, so only sintax works for it); the statistic the length probe needs is present in probe_length_stat.
  • Shotgun: the kraken_db directory exists when Kraken2 runs, and the sylph databases and their lineage tables are in place when sylph runs.
  • Both modes (in shotgun mode when Kraken2 runs): every keep/discard token of the taxon filter can match the run's taxonomy strings. A token written for MetaFlux v2.4.0 or earlier, such as k__Bacteria, stops the run with the token to write instead (d__Bacteria); see keeping and discarding taxa.

Failures are reported plainly, for example:

[MetaFlux] no *_1.* FASTQs found in /path/to/fastq_dir
[MetaFlux] mode=amplicon requires {sample}_R1/_R2 naming; found {sample}_1/_2 in /path/to/fastq_dir
[MetaFlux] mode=amplicon requires primer FASTA but not found: /path/to/fwd_primer.fasta
[MetaFlux] mode=shotgun requires kraken_db but not found: /path/to/kraken2_db

The first message names the fallback pattern, _1, because MetaFlux looks for _R1/_R2 first and only reports the failure after the SRA-style names have also come up empty — it means neither convention matched anything in that directory.

Two things that check does not do. MetaFlux creates fastq_dir if it is missing, because BioProject mode downloads into it, so a mistyped path produces the no *_1.* FASTQs found message above rather than a missing-directory error. And mates are matched from R1 only — a missing R2 is reported later by Snakemake as a missing input file, not by MetaFlux.

The dry-run also prints a one-line summary of what MetaFlux understood; read it before committing cores to a run:

[MetaFlux v2.6.1] mode=amplicon, input=local, 24 sample(s) in /path/to/fastq_dir

A sample count that does not match the sequencing run usually means a naming mismatch — an unexpected mate suffix, or files sitting one directory deeper than fastq_dir. Below that summary, Snakemake lists the jobs it would run and how many times each. A job count of zero means everything is already up to date; large unexpected counts usually mean a config change invalidated an earlier stage.

Warning

The dry-run cannot catch everything. It checks paths, names and parameter validity — not whether mode matches the biology of the data. Both modes accept {sample}_R1/_R2 files, so amplicon reads pushed through shotgun mode (or the reverse) will run without complaint and produce a meaningless table. See choosing a mode.

Step 5 — the run

snakemake --sdm conda --cores 16 --configfile config/config.yaml

--cores is the total number of CPUs Snakemake may use at once across all concurrent jobs; the split between rules comes from the resources block in the config. The first run also builds the conda environments and downloads the reference databases the selected marker needs, so it is considerably slower than the second.

The run is resumable. If it is interrupted, repeating the same command picks up from the last completed step; add --rerun-incomplete if Snakemake reports outputs left in an incomplete state. Changing a parameter re-runs only the steps downstream of it — trying a different truncation length, for example, does not repeat primer trimming.

Where the results are

Everything lands under out_dir in numbered, stage-ordered directories. The two files most runs end at:

Mode Final table Read tracking
amplicon 06.taxonomy/asv_table.txt stats/read_tracking.txt
shotgun 03.abundance/otu_table.tsv (plus otu_table.biom) stats/read_tracking.txt

Both modes also write multiqc/multiqc_report.html, and read_tracking.txt shows how many reads survived each stage per sample — the first thing to look at when a table comes out thinner than expected. The full layout is in output files.

Next