My Work with the Penn-CHOP Microbiome Program

Hereunder is a list of some of the more presentable (and open source) projects I've worked on as a developer with the Penn-CHOP Microbiome Program. See also the new MicroBioCode website!
I've contributed to all these projects to varying degrees. A few I built from scratch, others I've updated and maintain, and to some I've just made minor additions. Use the buttons on each card to see my contributions (my GH username is Ulthran).

Status Dashboard

Metagenomics Pipelines

Sunbeam: a robust, extensible metagenomic sequencing pipeline

A pipeline written in snakemake that simplifies and automates many of the steps in metagenomic sequencing analysis. It uses conda and docker to manage dependencies and can be deployed on most Linux workstations and clusters. It is easily extensible to perform further analyses. To read more, check out the paper in Microbiome.
sbx_assembly

A Sunbeam extension for assembly of contigs using Megahit, gene annotation using Prodigal, and annotation using Blast and Diamond. It can also map reads to contigs and calculat per-base coverage using Minimap2 and samtools.
sbx_coassembly

A Sunbeam extension to perform co-assembly of reads from arbitrary groups of samples from a given project using Megahit.
sbx_mapping

A Sunbeam extension for mapping reads against reference genomes using bwa and performing custom filtering.
sbx_kraken

A Sunbeam extension for taxonomic assignment of reads to databases using Kraken. You can get pre-built kraken databases at the Kraken homepage or build your own then specify the path to your database in the config.
sbx_demic

A Sunbeam extension for estimating bacterial growth rates via peak-to-trough ratios (PTRs) using DEMIC. In preparation to use demic, reads are first assembled using Megahit, then binned by inferred genome using MaxBin2, after which reads are mapped back onto contigs using Bowtie2 and Samtools.
sbx_gene_clusters

A Sunbeam extension for reads-level alignment to gene clusters of interest, e.g. bai operon or butyrate producing genes.
sbx_genome_assembly

A Sunbeam extension for de novo microbial genome assembly. This pipeline uses SPAdes for single genome assembly and CheckM and read map coverage of the assembled genome for quality assessment (sbx_SPARC rules). In addition, this pipeline uses hmmer to identify SCCGs (sbx_SCCG rules).
sbx_virus_id

A Sunbeam extension that identifies, filters, and annotates viruses from metagenomic samples. It can be configured to use multiple different assemblers and virus identification softwares.
sbx_marker_magu

A Sunbeam extension for detecting and quantifying phages, bacteria, and archaea in whole genome shotgun reads from human-derived samples using Marker-MAGu.
sbx_mgv

A Sunbeam extension for classifying viral sequences using MGV.
sbx_seeker

A Sunbeam extension for discriminating between virus and phage sequences using the alignment-free Seeker deep learning algorithm.

Statistical Packages

DEMIC: dynamic estimator of bacterial groth rates

A dynamic estimator of microbial communities. It employs a multi-sample algorithm based on contigs and coverage values to infer the relative distances of contigs from the replication origin and to accurately compare bacterial growth rates between samples.
unassigner

Evaluates consistency with named bacterial species for short 16S rRNA marker gene sequences. This is achieved by ruling out taxonmic bins for each input 16S sequence rather than by trying to assign them to specific bins, avoiding certain pitfalls of the other method such as disparate matching bins.
ZIBR: Zero-Inflated Beta Random Effects Model

A two-part zero-inflated Beta regression model with random effects for testing the association between microbial abundance and clinical covariates for longitudinal microbiome data.
DAFOT: Detector of Active Flow On a Tree

The main goal of this package is to provide a new method for two-sample testing for microbial compositional data by leveraging the phylogenetic tree information. Empirical evidence from real data sets suggests that the phylogenetic microbial composition difference between two populations is usually sparse. Motivated by this observation, this package implements a new maximum type test that is particularly powerful against sparse phylogenetic composition differences and enjoys certain optimality.

Utility Libraries

ShotgunUniFrac

A dual use program for downloading and extracting genes from NCBI and for creating phylogenetic trees for many marker genes and merging the results into one. This was my first from-scratch project for the PCMP.
PyCov3

A package for generating cov3 files which are generated from sam files giving coverage information and a fasta file giving binned contigs. Cov3 files are used as input for the DEMIC R package which calculates PTR, an estimate for bacterial growth rates.
primertrim

Detect short primer sequences in FASTQ reads and trim the reads accordingly.
DNAbc

Identify DNA barcodes in FASTQ data files and write demultiplexed data.
heyfastq

A Python library for reading and writing FASTQ files.

Services

PCMP Metadata Checker

A Flask app used by bioinformaticians to ensure their metadata meet the standards of the PCMP. It also maintains a database of projects and their metadata sheet(s). I have experimented with numerous methods for deployment for this app including on PythonAnywhere, on a dedicated CHOP server with nginx, as a serverless app on AWS (similar to this site), and, the current method, as a containerized microservice in a CHOP hosted kubernetes cluster. Unfortunately for you it is hosted on Enterprise GH to take advantage of hosted CI/CD runners and its domain is restricted to CHOP systems.

DevOps Tools

Conda Env Check: a Conda environment health check workflow

A GitHub Action that can provide a number of health metrics for Conda environments including verifying that it is createable, verifying that any pinned versions are retrievable, and checking versions against latest and solved.
sbx_test_action

A GitHub Action that manages setup of the core Sunbeam pipeline for testing extensions in a CI environment. It automatically handles branching, forking, and user information to always test the correct version of any given extension.
sbx_template

A template repository that allows for quick and easy creation of new Sunbeam extensions. It includes workflows to automatically fill in the new extension name and organization in many files.