BCCDC-PHL Auto Illumina Run QC Check

Indices and tables

auto_illumina_run_qc_check.config

This module includes the functionality for loading application configuration.

auto_illumina_run_qc_check.config.load_config(config_path)

Load the application config file.

Parameters:

config_path (PathLike) – Path to config file.

Return type:

Config

Returns:

A dictionary containing configuration data.

auto_illumina_run_qc_check.core

This module includes the core functionality for performing QC checks on an illumina sequencing run.

auto_illumina_run_qc_check.core.find_run_dirs(config, check_upload_complete=True)

Find sequencing run directories under the ‘run_parent_dirs’ listed in the config.

Parameters:

  • config (Config) – Application config.

  • check_upload_complete (bool) – Check for presence of ‘upload_complete.json’ file.

Return type:

Iterator[Optional[dict]]

Returns:

Run directory, or None. Keys: [‘sequencing_run_id’, ‘path’, ‘instrument_type’]

auto_illumina_run_qc_check.core.get_sum_sample_fastq_file_sizes(run)

Get the sum of all sample fastq file sizes in the run directory.

Parameters:

run (dict) – Run directory. Keys: [‘sequencing_run_id’, ‘path’, ‘instrument_type’]

Return type:

float

Returns:

Sum of all sample fastq file sizes in the run directory.

auto_illumina_run_qc_check.core.qc_check(config, run)

Initiate an analysis on one directory of fastq files.

Parameters:

  • config (Config) – Application config.

  • run (dict) – Run directory. Keys: [‘sequencing_run_id’, ‘path’, ‘instrument_type’]

Return type:

Optional[dict]

Returns:

The QC check results

auto_illumina_run_qc_check.core.scan(config)

Scanning involves looking for all existing runs and storing them to the database, then looking for all existing symlinks and storing them to the database. At the end of a scan, we should be able to determine which (if any) symlinks need to be created.

Parameters:

config (Config) – Application config.

Return type:

Iterator[Optional[dict]]

Returns:

A run directory to analyze, or None

auto_illumina_run_qc_check.instrument

This module includes functionality related to determining the instrument type of a given run.

auto_illumina_run_qc_check.instrument.determine_instrument_type(run_id)

Determine the instrument type

Parameters:

run_id (str) – The sequencing run ID.

Return type:

InstrumentType

Returns:

The instrument type

auto_illumina_run_qc_check.logging_config

This module includes functionality related to configuring application logging.

class auto_illumina_run_qc_check.logging_config.JSONFormatter(fmt=None, datefmt=None, style='%', validate=True, *, defaults=None)

Custom formatter for logging in structured JSON Lines.

format(record)

Format a log record as a JSON object, encoded as a string.

Parameters:

record (LogRecord)

Return type:

str

formatTime(record, datefmt=None)

Returns the creation time of the LogRecord formatted with milliseconds. :type record: LogRecord :param record: The logging record

Return type:

str

Returns:

The log record creation time, formatted as an ISO timestamp

Parameters:

datefmt (Optional[str])

auto_illumina_run_qc_check.logging_config.configure_logging(log_level='info')

Configure logging

Parameters:

log_level (str) – Log level (‘debug’, ‘info’, ‘warning’, ‘error’) default: ‘info’

auto_illumina_run_qc_check.model

This module includes classes used to represent domain entities in a type-safe way.

class auto_illumina_run_qc_check.model.Config(scan_interval_seconds=3600, notification=<factory>, qc_thresholds=<factory>, run_parent_dirs=<factory>, excluded_runs_list=None, excluded_runs=<factory>, projects_definition_file=None, projects=<factory>)

Main application config.

Parameters:

  • scan_interval_seconds (int)

  • notification (dict)

  • qc_thresholds (list[dict])

  • run_parent_dirs (list[Path])

  • excluded_runs_list (Optional[Path])

  • excluded_runs (list[str])

  • projects_definition_file (Optional[Path])

  • projects (list[dict])

class auto_illumina_run_qc_check.model.InstrumentType(*values)

Instrument type (‘miseq’, ‘nextseq’, ‘i100’ or ‘unknown’)

auto_illumina_run_qc_check.notification

This module includes functionality related to sending notification emails.

auto_illumina_run_qc_check.notification.send_notification_email(run_dir, notification_config)

Collect relevant data from an analysis output dir and send

Parameters:

  • run_dir (PathLike) – Sequencing run output dir (must include a “qc_check_complete.json” file).

  • notification_config (dict) – Notification-related config. Required keys: [‘auth_url’, ‘email_url’, ‘client_id’, ‘client_secret’, ‘sender_email’]

Return type:

Optional[dict]

Returns:

Response data, or None if authentication fails.

auto_illumina_run_qc_check.parsers

This module includes functions for parsing the interop summary and RunParameters.xml files

auto_illumina_run_qc_check.parsers.parse_interop_summary(summary_lines)

Parse an interop summary csv file into a dict.

Parameters:

summary_lines (list[str]) – A list of lines from an interop summary csv file.

Returns:

A dict containing the parsed interop summary. Keys: [‘ClusterDensity’, ‘ErrorRate’, ‘IntensityCycle1’, ‘PercentAligned’, ‘PercentGtQ30’, ‘ProjectedTotalYield’, ‘YieldTotal’, ‘Reads’, ‘LanesByRead’]

Return type:

dict[str, object]

auto_illumina_run_qc_check.parsers.parse_lanes_by_read(summary_lines)

Parse a read summary csv file into a list of dicts.

Parameters:

summary_lines (list[str]) – A list of lines from a read summary csv file.

Returns:

A list of dicts containing the parsed read summary. Keys: [‘ReadNumber’, ‘LaneNumber’, ‘Surface’, ‘TileCount’, ‘Density’, ‘DensityDeviation’, ‘PercentPf’, ‘PercentPfDeviation’, ‘Reads’, ‘ReadsPf’, ‘PercentGtQ30’, ‘Yield’, ‘CyclesError’, ‘PercentAligned’, ‘PercentAlignedDeviation’, ‘ErrorRate’, ‘ErrorRateDeviation’, ‘ErrorRate35’, ‘ErrorRate35Deviation’, ‘ErrorRate75’, ‘ErrorRate75Deviation’, ‘ErrorRate100’, ‘ErrorRate100Deviation’, ‘IntensityCycle1’, ‘IntensityCycle1Deviation’, ‘PhasingSlope’, ‘PhasingOffset’, ‘PrePhasingSlope’, ‘PrePhasingOffset’, ‘ClusterDensity’, ‘Occupancy’]

Return type:

list[dict[str, object]]

auto_illumina_run_qc_check.parsers.parse_read_line(read_line, read_number)

Parse a line from a read summary csv file into a dict.

Parameters:

  • read_line (str) – A line from a read summary csv file.

  • read_number (int) – The read number.

Returns:

A dict containing the parsed read line. Keys: [‘ReadNumber’, ‘LaneNumber’, ‘Surface’, ‘TileCount’, ‘Density’, ‘DensityDeviation’, ‘PercentPf’, ‘PercentPfDeviation’, ‘Reads’, ‘ReadsPf’, ‘PercentGtQ30’, ‘Yield’, ‘CyclesError’, ‘PercentAligned’, ‘PercentAlignedDeviation’, ‘ErrorRate’, ‘ErrorRateDeviation’, ‘ErrorRate35’, ‘ErrorRate35Deviation’, ‘ErrorRate75’, ‘ErrorRate75Deviation’, ‘ErrorRate100’, ‘ErrorRate100Deviation’, ‘IntensityCycle1’, ‘IntensityCycle1Deviation’, ‘PhasingSlope’, ‘PhasingOffset’, ‘PrePhasingSlope’, ‘PrePhasingOffset’, ‘ClusterDensity’, ‘Occupancy’]

Return type:

dict[str, object]

auto_illumina_run_qc_check.parsers.parse_read_summary(summary_lines)

Parse a read summary csv file into a list of dicts.

Parameters:

summary_lines (list[str]) – A list of lines from a read summary csv file.

Return type:

list[dict]

Returns:

A list of dicts containing the parsed read summary. Keys: [‘ReadNumber’, ‘IsIndexed’, ‘TotalCycles’, ‘YieldTotal’, ‘ProjectedTotalYield’, ‘PercentAligned’, ‘ErrorRate’, ‘IntensityCycle1’, ‘PercentGtQ30’]

auto_illumina_run_qc_check.parsers.parse_read_summary_line(read_summary_line)

Parse a line from a read summary csv file into a dict.

Parameters:

read_summary_line (str) – A line from a read summary csv file.

Return type:

dict

Returns:

A dict containing the parsed read summary line. Keys: [‘ReadNumber’, ‘IsIndexed’, ‘TotalCycles’, ‘YieldTotal’, ‘ProjectedTotalYield’, ‘PercentAligned’, ‘ErrorRate’, ‘IntensityCycle1’, ‘PercentGtQ30’]

auto_illumina_run_qc_check.parsers.parse_run_parameters_xml(run_parameters_xml_path, instrument_type)

Parse a run parameters xml file into a dict.

Parameters:

  • run_parameters_xml_path (PathLike) – The path to the run parameters xml file.

  • instrument_type (str) – The instrument type. One of [‘miseq’, ‘nextseq’].

Returns:

A dict containing the parsed run parameters. Keys: [‘flowcell_version’]

Return type:

dict[str, object]

auto_illumina_run_qc_check.parsers.parse_run_stats(summary_lines)

Parse a run stats csv file into a dict.

Parameters:

summary_lines (list[str]) – A list of lines from a run stats csv file.

Returns:

A dict containing the parsed run stats. Keys: [‘PercentGtQ30’, ‘ProjectedTotalYield’, ‘YieldTotal’, ‘ErrorRate’, ‘PercentAligned’, ‘Occupancy’, ‘Reads’]

Return type:

dict[str, object]

auto_illumina_run_qc_check.samplesheet

This module includes functionality related to parsing illumina SampleSheet.csv files.

auto_illumina_run_qc_check.samplesheet.find_samplesheet_path(run_dir)

Given a run directory path, find the path to the SampleSheet.csv file that can be used to summarize num samples by project ID.

Parameters:

run_dir (Path) – Path to the run directory

Return type:

Optional[Path]

Returns:

Path to the SampleSheet.csv file, or None if not found.

auto_illumina_run_qc_check.samplesheet.parse_samplesheet(samplesheet_path, instrument_type)

Parse a SampleSheet, given the path to the SampleSheet file and the Instrument type.

Parameters: