Here, we present a computational protocol to perform a spatiotemporal reconstruction of an epidemic. We describe steps for using epidemiological data to depict how the epidemic changes over time and for employing clustering analysis to group geographical units that exhibit similar temporal epidemic progression. We then detail procedures for analyzing the temporal and spatial dynamics of the epidemic within each cluster. This protocol has been developed to be used on historical data but could also be applied to modern epidemiological data.For complete details on the use and execution of this protocol, please refer to Galli et al. (2023).  •Death-based clustering for epidemic reconstruction•Protocol suitable for historical and modern epidemiological data analysis•Spatial dynamics analysis of the epidemic on geographical maps Publisher’s note: Undertaking any experimental protocol requires adherence to local institutional guidelines for laboratory safety and ethics. Here, we present a computational protocol to perform a spatiotemporal reconstruction of an epidemic. We describe steps for using epidemiological data to depict how the epidemic changes over time and for employing clustering analysis to group geographical units that exhibit similar temporal epidemic progression. We then detail procedures for analyzing the temporal and spatial dynamics of the epidemic within each cluster. This protocol has been developed to be used on historical data but could also be applied to modern epidemiological data. The protocol below shows a way for the use of historical data (such as death records) to reconstruct the spatiotemporal evolution of a past epidemic. As an example, we analyzed the information contained in XVII century death registers of the city of Milan  to study the spatial and temporal diffusion of a plague epidemic inside the city.This section includes the minimal software and hardware requirements, the installation procedures, as well as the format of the files to be processed throughout this protocol.Timing: 30 min1.https://cran.r-project.org/ (Note that the protocol has been tested on R version 4.1.3 (2022-03-10)).R is a freely available language and environment for statistical computing and graphics.2.https://www.rstudio.com/.RStudio is a user-friendly integrated development environment for R.Optional: QGIS is a free and open-source geographic information systemhttps://www.qgis.org/en/site/.Timing: 10 min3.To run this protocol, it is required to previously install the R packages present in the “> install.packages(“name_of_the_package”)Timing: 10 min–1 hYou can follow the protocol using your data or our sample dataset available on GitHub.4.https://github.com/RiccardoND/STAR_protocol_epidemic_reconstruction (GitHub: https://doi.org/10.5281/zenodo.8214153) or directly clone the GitHub repository (∼10 Mb), by running the following command on your terminal.You can download our sample dataset from https://github.com/RiccardoND/STAR_protocol_epidemic_reconstruction.> git cloneNote: This repository includes all the data and code necessary to reproduce the protocol. In this dataset, each row contains the total number of “cases” (or deaths) for each cause of death, for each day (or other unit of time) and for each spatial location.5.Format the table to obtain a single case in each row.> df <- read.csv> tab <- data.frame)> tab$count <- NULL> tab$Date <- as.Date(tab$Date)> tab <- tab #sort by dateNote: It is always possible to format your dataset in other ways or add other information, but the protocol must be adjusted accordingly.Note: a version of the dataset already formatted as described in this step is available in the file TableS1_formatted.csv.To follow the protocol step-by-step, you must have the data formatted in a dataset in which each row contains a single case. If you have a dataset with cumulative numbers, see below on how to format it properly for the protocol.Timing: 20–30 min1.Prepare the workspace directory. Open RStudio and set the working directory path.> setwd("yourpath")2.Load the dataset of cases/deaths and related information and format it as shown in step 5 of “> head(tab)3.Create a time-series plot that shows the progression of the daily number of deaths for each cause of death (in our case Plague or not plague) .Figure 1> library(ggplot2)>library(tidyverse)> tab %>% group_by %>% summarize(count = n) %>% ggplot) + geom_line(size = 0.7) + geom_point(size = 2) + theme_bw + scale_color_manual) + theme,  axis.title.x = element_text(size = 15),  axis.title.y = element_text(size = 15)) + labs(y = 'Number of Deaths')Note: The plot compares the progression of deaths caused by the disease of interest to deaths related to any other cause: it could show the presence of any kind of seasonality, and the completeness of the dataset. The absence of a signal in the period between 4 and 30 August is due to reasons not related to the analysis.To reconstruct the dynamics of an epidemic, it is helpful to start by plotting the number of deaths (or cases) that occurred at a specific time. In this case, we are going to build a time-series plot with two lines, one for deaths unrelated to the disease of interest, i.e., plague, and one for deaths related to plague. This step allows us to depict the temporal progression of the epidemic in the city and to detect any period with missing data or any other anomaly.Timing: 3–4 h4.Load the R packages in the current RStudio session.> library(tidyverse)> library(reshape2)> library(factoextra)> library(ade4)> library(vegan)> library(RColorBrewer)> library(inflection)>library(ggpubr)5.Load the table with one row per case, as shown in step 5 in the “6.a.> tab_1630_peste <- droplevels)> head(tab_1630_peste)filter for the cause of death or disease of interest.b.> t_1630_peste <- as.matrix)build a matrix in which each column is a parish , and each row is a day .c.> days1630 <- seq(from=as.Date("1630-01-01"), to=as.Date("1630-12-31"), by=1)> absent_days <- as.Date)), origin="1970-01-01")> t_1630_peste_absent <- matrix(ncol=ncol(t_1630_peste), nrow=length(absent_days))> t_1630_peste_absent[is.na(t_1630_peste_absent)] <- 0> row.names(t_1630_peste_absent) <- as.character(absent_days)> t_1630_peste_all <- rbindNote: Select your temporal period of interest. In the example dataset, the cases span the year 1630, thus our range is from the 1st of January 1630 to the 31st of December of the same year.Add to the matrix the days in which there are no recorded plague deaths.d.> t_1630_peste_all_ord <- t_1630_peste_allOrder the table chronologically.e.> peste_cum <- matrix, nrow=0)> for ) { tmp <- colSums peste_cum <- rbind}> peste_cum <- rbind> row.names(peste_cum) <-row.namescreate a cumulative matrix.f.> peste_cum_norm <- apply x/max(x))Normalize the number of deaths by dividing the number of daily deaths in each parish by the total number of deaths in that parish .g.> peste_cum_norm_m <- melt(peste_cum_norm)> colnames(peste_cum_norm_m) <- )> peste_cum_norm_m$Date <- as.Date(peste_cum_norm_m$Date)> Cumulative_curves <- ggplot) + geom_line + theme_bw + ylab("Cumulative relative frequency of plague deaths (%)")Note: Some parishes may not have enough cases to build a reliable cumulative curve on the selected time range. We can choose a threshold under which the parishes will be excluded from the subsequent analyses. For example, we selected only the parishes with more than one plague death every two weeks during the epidemic.Plot the cumulative relative frequency curves of the parishes plague deaths .> peste_h.Note: The epidemic period is not simply the time between the first and last case, in fact, as in our data, there may be some isolated cases before or after the epidemic. Thus, it is important to analyze the epidemic curve. In our data, the epidemic begins around the middle of March and continues throughout 1630.Calculate the duration of the epidemic. To do so, we have to observe the epidemic curve produced in step 3 to manually determine the starting and ending period of the epidemic.i.> weeks_of_epidemic <- 42> death_count_thr <- weeks_of_epidemic/2#21Calculate the death threshold. To select parishes with more than 1 death every two weeks, we have to look for parishes with more than 21 deaths (week of the epidemics / 2), as explained in Galli et al., 2023.j.> parr_sel <- peste_cum > death_count_thr> peste_cum_norm_sel <- peste_cum_norm> peste_cum_norm_melt <- melt(peste_cum_norm_sel)> colnames(peste_cum_norm_melt) <- c> peste_cum_norm_melt$Date <- as.Date(peste_cum_norm_melt$Date)> cumulative_curves_sel <- ggplot) + geom_line + scale_color_manual) + theme_bw + ylab("Cumulative relative frequency of plague deaths (%)")Remove parishes with fewer cases than the threshold and plot only the selected cumulative curves .> parr_sBuild the cumulative curves of the plague deaths for each parish .7.Note: To cluster the parishes (or geographic units) in two or more groups, we are going to perform a k-means clustering on the result of the Principal Coordinates Analysis (PCoA) performed on the cumulative curves of plague deaths. In particular, we are going to compare the cumulative curves of the different parishes with each other to generate a distance matrix, which will be subjected to PCoA. Performing a PCoA on the dataset allows us to reduce the dimension of our data and to visualize them in two (or three) dimensions. Then we can apply k-means clustering, an unsupervised clustering algorithm that groups a dataset into a specific number of clusters . The k-means algorithm will assign each observation (in our case each parish) to a cluster on the basis of their position on the PCoA space.,a.> dist <- dist(t(peste_cum_norm_sel))Calculate the Euclidean distance matrix between the cumulative curves of the different parishes.b.> pcoa <- cmdscale> plot(pcoa$eig)> pcoa <- cmdscaleNote: First, we must determine the best number of axes for the PCoA analysis on the basis of the eigenvalues. The plot shows that the eigenvalues drastically drop down for the first three dimensions.Perform the Principal Coordinates Analysis (PCoA) and make the scree plot .> pcoa <c.> fviz_nbclustNote: We determined the optimal number of clusters in the dataset using a popular cluster validation index: the average silhouette width method.,Determine the optimal number of clusters in which the parishes (or geographic units) can be divided .> fviz_nd.> f <- kmeans> clusters <- as.matrix(f$cluster)> clus <- clustersClusteringPerform the clustering analysis on the basis of the cumulative curves.8.Use the PERMANOVA test to determine if the separation between the clusters is statistically significant.> a <- adonis2> p_value <- a$`Pr(>F)`[1]> p_value[1] 0.0019.Plot the results of the PCoA and the k-means clustering .Figure 6> plot_pcoa <- s.class,   col = c,   cellipse = 0,   sub = paste,   xlim = c)In this step, we are going to group cases/deaths based on the progression of the epidemic. In particular, we are going to analyze the specific cumulative epidemiological curves of each geographical unit. In the example dataset about the 1630 plague epidemic in Milan, the geographical units are the parishes in which the city was divided and where the person died . Other geographical units can be streets, houses, villages, districts, etc.Timing: 3–4 h10.Load the necessary R packages in the current RStudio session.> library(RColorBrewer)> library(inflection)> library(ggpubr)11.Color the cumulative relative frequency curves of plague deaths for each parish on the basis of their clusters .Figure 7> peste_cum_norm_melt <- melt(peste_cum_norm_sel)> colnames(peste_cum_norm_melt) <- c> clusters <- as.matrix(f$cluster)> peste_cum_norm_melt$Date <- as.Date(peste_cum_norm_melt$Date)> peste_cum_norm_melt$Cluster <- as.character(clusters)> Cumulative_curves_sel_clusters <- ggplot) + geom_line + scale_color_manual) + theme_bw + ylab("Cumulative relative frequency of plague deaths (%)")12.Save a table with the information about the clusters and the corresponding parishes to be used later for further analysis on the clusters.> palette <- c> tab_clusters <- data.frame(Parish = row.names(clusters), Cluster = > as.data.frame(clusters)$V1, Color = palette[as.matrix(clusters)])> head(tab_clusters)> write.csv13.Note: As an example, we are going to determine: the first plague case for each parish, the inflection points of the cumulative curves, and the date at which the parishes of the two clusters reached 25%, 50%, 75%, and 100% of their total plague deaths.a.> peste_cum_norm_melt_first <- peste_cum_norm_melt> peste_cum_norm_melt_first_nodup <- peste_cum_norm_melt_first> peste_cum_norm_melt_first_nodup <- NULL> colnames(peste_cum_norm_melt_first_nodup) <- cCalculate the dates on which the parishes of the two clusters reached the first plague death.b.> peste_cum_norm_melt_25 <- peste_cum_norm_melt> peste_cum_norm_melt_25_nodup <- peste_cum_norm_melt_25> peste_cum_norm_melt_25_nodup <- NULL> colnames(peste_cum_norm_melt_25_nodup) <- cCalculate the dates on which the parishes of the two clusters reached 25% of total plague deaths.c.> peste_cum_norm_melt_50 <- peste_cum_norm_melt> peste_cum_norm_melt_50_nodup <- peste_cum_norm_melt_50> peste_cum_norm_melt_50_nodup <- NULL> colnames(peste_cum_norm_melt_50_nodup) <- cCalculate the dates on which the parishes of the two clusters reached 50% of total plague deaths.d.> peste_cum_norm_melt_75 <- peste_cum_norm_melt> peste_cum_norm_melt_75_nodup <- peste_cum_norm_melt_75> peste_cum_norm_melt_75_nodup <- NULL> colnames(peste_cum_norm_melt_75_nodup) <- cCalculate the dates on which the parishes of the two clusters reached 75% of total plague deaths.e.> peste_cum_norm_melt_100 <- peste_cum_norm_melt> peste_cum_norm_melt_100_nodup <- peste_cum_norm_melt_100> peste_cum_norm_melt_100_nodup <- NULL> colnames(peste_cum_norm_melt_100_nodup) <- cCalculate the dates on which the parishes of the two clusters reached 100% of total plague deaths.f.> infl_date_tab <- matrix)> colnames(infl_date_tab) <- c> for (i in 1:ncol(peste_cum_norm)){ col = colnames(peste_cum_norm)[i] infl_date <- as.Date(bede(as.numeric(as.Date(row.names(peste_cum_norm))), peste_cum_norm,0)$iplast, origin = "1970-01-01") infl_date_tab <- as.character(infl_date) infl_date_tab <- col}Calculate the dates on which the cumulative curves of the parishes of the two clusters change concavity, corresponding to the epidemic peak .g.> all_tab_tmp <- merge> colnames[1:2] <- c> all_tab_tmp1 <- merge> all_tab_tmp2 <- merge> all_tab_tmp3 <- merge> all_tab_tmp4 <- merge> all_tab <- merge> head> all_tab$Inflection_date <- as.Date> all_tab2 <- melt)Merge all the data in one table.h.> all_tab2$Cluster <- factor, ordered = TRUE)> my_comparisons <- list)> labels <- list#label name of facet> labels <- list> facet_labeller <- function{return}> Clusters_boxplot <- ggboxplot + scale_fill_manual) + geom_jitter) + facet_wrap + stat_compare_means + theme(legend.position = "none") + ylab( "Date") + theme(strip.text.x = element_text(size = 8))Plot the results as boxplots and determine if the differences between the two clusters are statistically significant > gps <- read.csv> clusters <- read.csv("Clusters.csv")Load the table with the epidemiological data, the table with the GPS information about the geographic units, in this case, the parishes, and the table with the clusters (see step 12).b.> df2 <- data.frame)> df3 <- df2 %>% select(-count)> tab <- left_join> tab$Date <- as.Date(tab$Date)> tab_cluster <- left_join> tab_cluster_2 <- droplevels))#drop rows without cluster total number of cases in the 2 clusters = 7002Produce a summary that integrates all the information for your dataset.c.> tab_cluster_2$Weeks <- as.numeric)> tab_cluster_2$Cluster <- factor)> tab_cluster_3 <- tab_cluster_2 %>% filter(Death_cause == "Plague")> cc_clusters <- tab_cluster_3 %>% group_by %>% summarize(count = n) %>% ggplot) + geom_line(size = 0.7) + geom_point(size = 2) + theme_bw + scale_color_manual) + theme(axis.title.x = element_text(size = 12),  axis.title.y = element_text(size = 12)) + labs(y = 'Number of Deaths')Note: The number of color values assigned to the function “scale_fill_manual” must be the same as the number of clusters.Plot the weekly number of plague deaths for each cluster .> tab_clVisualize the epidemiological evolution of the epidemic in the parishes of the two clusters.The parishes, and therefore our cases, have been clustered on the basis of the temporal progression of the epidemic. Now we can analyze what are the differences and similarities between the clusters.Timing: 4 h15.Load the R packages in the current RStudio session.> library(png)> library(grid)16.Produce a summary table that integrates all the information for your dataset (see step 14b).> head(tab_cluster)# A tibble: 6 x 7# Groups: Parish, Cluster, Color, Death_cause, Latitude [6]Note: Latitude and Longitude are the columns used to indicate geographic coordinates.17.Filter the dataset to remove all the parishes without geographic information.> df <- tab_cluster %>% filter(!is.na(Latitude))18.Create a new gray cluster (Cluster “0”) for the unassigned parishes .> df$Cluster[is.na(df$Cluster)] <- 0> df$Color[is.na(df$Color)] <- "gray"19.Summarize the number of cases for each parish, cluster, death cause, and coordinates.> df2 <- df %>% group_by %>%summarize(Count = n)> head(df2)# A tibble: 6 × 7# Groups: Parish, Cluster, Color, Death_cause, Latitude [6]20.Clean the table removing non-plague-related deaths.> df2$Cluster <- factor,ordered = TRUE)> peste_clus_gps <- df2 %>% filter(Death_cause == "Plague")> peste_clus_gps$Death_cause <- NULL> peste_clus_gps <- peste_clus_gps> peste_clus_gps$Latitude <- as.numeric(peste_clus_gps$Latitude)> peste_clus_gps$Longitude <- as.numeric(peste_clus_gps$Longitude)> peste_clus_gps$Count <- as.numeric(peste_clus_gps$Count)21.png image and transform it into a raster image graphical object.Load your map stored as a > map <- readPNG("positron_darker_2023.png")> map_2_plot <- rasterGrob22.Annotate the GPS coordinates of the four corners of the map image.Note: we can retrieve our base map image from QGIS by cropping the area of interest and annotating the GPS coordinates of the margins of our crop. This is essential to plot the points in their exact location on the map: the map itself needs to be referenced to the real GPS coordinates so that the point can be plotted using the real GPS coordinates available for each parish.> gps_map <-data.frame,   Y = c,   fid = c,   crop = c)> xmin <- gps_map #Bottom Left margin> ymin <- gps_map #Bottom Right margin> xmax <- gps_map #Top Right margin> ymax <- gps_map #Top Right margin23.Find the aspect ratio of the png image and use the “zoom” variable to scale the output file size.> img_width <- ncol(map)> img_height <- nrow(map)> aspect_ratio <- img_width/img_height> zoom <- 15Note: With the aspect ratio of the map and a multiplicative factor (zoom variable), it is possible to save the final map image at different sizes maintaining good resolution. The reasonable value of the zoom variable depends greatly on the size (in pixels) and the shape of the initial crop of the map.24.Plotting the parishes over the map according to the GPS coordinate. The size of the points is associated to the number of plague deaths and the color represents the cluster of the parishes.Note: The aspect ratio of the map must be maintained. To do so, we have to fix the width and height of the final plot file. RStudio may automatically plot the map with an incorrect aspect ratio. We strongly recommend saving the png file using the provided commands instead.> p <-ggplot)+ annotation_custom+ geom_point)+ scale_color_manual)+ labs + xlim+ ylim+ theme_classic+ theme,  legend.title=element_text(size=zoom/2),  axis.ticks=element_blank,  axis.title.x=element_blank,  axis.title.y=element_blank,  panel.grid.major = element_blank,  panel.grid.minor = element_blank,  panel.background = element_blank,  legend.position = c )25.png and scale it using the zoom variable Generate a visualization of the distribution of the parishes of the different clusters on a map using the coordinates and the information about the parishes.Parishes localization on the map of the city of Milan. In blue, parishes of cluster 1; in red, those of cluster 2; in gray, parishes with less than 21 total plague deaths . The size of the points represents the total number of deaths related to the plague experienced by the parish.This protocol has been designed to reconstruct the epidemiological dynamics of an epidemic using the recorded deaths and their geographical location.The first outcome consists of a time-series plot where it is possible to compare the temporal evolution of the epidemics against the incidence of deaths unrelated to the disease of interest .Then, the protocol performs a clustering analysis on the geographical units  on the basis of their cumulative relative frequency of plague deaths. The final outcome of this step is the Principal Coordinates Analysis (PCoA) plot, where the parishes have been colored on the basis of the clusters .Once we find the clusters, we can start to analyze the temporal dynamics of the epidemic in each of them; in Lastly, we analyze the spatial dynamics clusters by visualizing the geographical position of the parishes on a map: The clustering analysis used in this protocol does not rely on any kind of geographic information. Although this approach is advantageous when we are dealing with historical data for which geographic information is rarer or imprecise, this approach may be limiting for the analysis of datasets in which this information is available and reliable. In this case, the clustering analysis should consider the implementation of geographic information.In step 5 of “Open the csv file with a spreadsheet application .In this application, we can easily apply any adjustment and export the spreadsheet as an xlsx or a csv file. The file should be correctly formatted and ready to be imported in R.> df <- read.csvFor xlsx files (Excel format file):> install.packages("readxl")> library(readxl)> df <- read_xlsx("TableS1.xlsx")For csv files, we can import the table in R using the same lines of code in step 5 of “At step 5 in the “> install.packages("vroom”)> library(vroom)> library(tidyverse)> df <- vroom> tab <- as_tibble)> tab$count <- NULL> tab$Date <- as.Date(tab$Date)> tab <- tab #sort by dateWhen a table is too large , the functions “read.csv” (for “csv files”) or “read.delim” (for tab delimited files) may take too much computational time and RAM to maintain the table in the environment. In this case, the “tibble” class can help reduce time and RAM needed to handle the dataset. Thus, to upload a large table it is better to use another R library such as “vroom”. The following code must be substituted to step 5 of the “At step 6, parishes name contains spelling errors.> as.matrix(names(table(tab$Parish)))It is possible that the dataset contains spelling errors, or the same parish written in different ways . In these cases, R does not consider them as the same parish. To find possible errors we can list all the parishes and manually check for errors.> tab$Parish <- gsubThen we can correct them using R. As an example, consider a situation in which the parish of “S. Bartolomeo” is written in two different ways: “S. Bartolomeo” and “S. bartolomeo”.In this way we substituted all the cells containing "S. bartolomeo" with "S. Bartolomeo".At step 7c, the silhouette analysis indicates an optimal number of clusters higher than two.The protocol has been developed on the example dataset in which silhouette analysis finds that two clusters is the optimal number to classify the parishes. For this reason, this protocol can be directly applied only when the two clusters are found to be optimal.Whenever the protocol finds that more than two clusters are needed to describe the dataset, the user should slightly modify the commands to be able to compare the clusters .In the section “GPS information is not always available, particularly when dealing with old datasets that may refer to particular geographic locations  that changed names or disappeared over time.The clustering approach applied in this protocol does not rely on any type of geographical information. Thus, you can follow the protocol from step 1 till step 14 without any specific geographic information .Further information and requests for resources and reagents should be directed to and will be fulfilled by the lead contact, Riccardo Nodari (This study did not generate new unique reagents.


The software and firmware are designed for Opal Kelly FPGA modules, yet the Python developments are generally useful to organize communication with peripheral chips.We are developing a data acquisition system (DAQ) for real-time feedback that uses FPGA-based control of and acquisition from various electronic chips, or peripherals. Because these peripherals communicate over multiple protocols  through an FPGA, we designed pyripherals retains as a dictionary key. When passed the location of the data field, parameterized functions automatically format the data or command as required by the communication interface used by the peripheral. The assembled message is passed to the appropriate hardware controller responsible for low-level communication with the peripheral. In this solution, the addressing, bit indexing, and formatting are handled by pyripherals before the message is sent over the Opal Kelly FrontPanel API to a hardware-level communication controller on an Opal Kelly FPGA  for rapid readout of photon return time histograms. Similar to pyripherals, the Python package registerMap  = the bit index of the upper end of the data field in the register. Ex. in a 32-bit register where the data field is located in the last 4 bits of the register, Bit Index (High) would be 31.Bit Index (Low) = the bit index of the lower end of the data field in the register. Ex. in a 32-bit register where the data field is located in the last 4 bits of the register, Bit Index (Low) would be 28.pyripherals then reads the spreadsheet, referred to as a register index in the documentation, and returns a dictionary of name-Register pairs using the Register.get_chip_registers static method. The example below retrieves the register index from the table above.»> MYADC_regs = Register.get_chip_registers(‘MyADC’)Register object holds all values from the spreadsheet for the data field it represents. A guide for creating a register index is located in the documentation. An example register is shown below.Each »> print(MYADC_regs[‘RESULT’])0×0[0:11]»> MYADC_regs[‘RESULT’].__dict__{‘address’: 0, ‘default’: 15, ‘bit_index_high’: 11, ‘bit_index_low’: 0, ‘bit_width’: 12}register abstraction of pyripherals allows user code to refer to data fields using only their names. The spreadsheet organization of data fields allows for user-friendly editing and sharing of data field information without the need to change user code. Specific applications include communicating with microcontrollers or development boards like Arduino as well as accessing data using SPI or I2C controllers.The pyripherals uses registers to assemble messages for SPI  and a bit_width of 32 that adds 7 to the address every time it is advanced.// bit_width=32 addr_step=7`define MYADC_WRITE_IN_GEN_ADDR 8’h04 WRITE_IN is data from the host computer into the FPGA that is destined for the ADC chip. More information on the syntax and meaning of these lines is available in the endpoint definitions guide.The naming convention for endpoints that contain addresses or bit indices is demonstrated below with curly brackets {} indicating placeholders to be completed by the user. Endpoint directions are from the perspective of the FPGA so Addresses:`define {CHIPNAME}_{ENDPOINT_NAME}{_GEN_ADDR} {hexadecimal address} // bit_width={bit_width} addr_step={addr_step}Bit Indices:`define {CHIPNAME}_{ENDPOINT_NAME}{_GEN_BIT} {decimal bit index} // addr={address or endpoint name} bit_width={bit_width}Note: the above comments must be placed on the same line as the ‘define. They are split here for readability.create_chips method which instantiates a specified number of chips, incrementing the endpoint addresses and bit indices according to the GEN_ADDR, GEN_BIT, bit_width, and addr_step parameters above.For multiple units of the same chip, each chip class has a endpoint.get_chip_endpoints which returns a dictionary of name-Endpoint pairs. An example using the “MYADC_WRITE_IN” endpoint from earlier is shown below.Once created, the user can read the endpoint definitions file with »> MYADC_eps = Endpoint.get_chip_endpoints(chip_name=’MYADC’)»> print(MYADC_eps[‘WRITE_IN’])0×4[None:None]»> MYADC_eps[‘WRITE_IN’].__dict__{‘address’: 4, ‘bit_index_low’: None, ‘bit_index_high’: None, ‘bit_width’: 32, ‘gen_bit’: False, ‘gen_address’: False, ‘addr_step’: 7}endpoint class in pyripherals extends the capabilities of the Opal Kelly FrontPanel API by automatically linking the Python and Verilog endpoint data with a shared definitions file. With pyripherals, when the user changes the value of an endpoint in the definitions file the change is reflected in both the Python and Verilog code.The pyripherals is available at https://github.com/lucask07/covg_fpga/. It is written for the Opal Kelly XEM7310 FPGA and supports I2C, SPI, and LVDS communication with a DDR for data buffering. An example use of this code is an impedance analyzer using a DAC80508 digital-to-analog converter (Our FPGA code for use with onverter  and an Aonverter  communichttps://pyripherals.readthedocs.io/en/latest/index.html and the GitHub is available at https://github.com/Ajstros/pyripherals. pyripherals is available for install from pip at https://pypi.org/project/pyripherals/.Documentation is available at pyripherals was developed under an NIH-funded project to create a digital ion channel amplifier at the University of St. Thomas where it is being used to communicate with and control an FPGA-based data acquisition system for real-time feedback.


Endoscopic submucosal dissection (ESD) as a treatment for superficial pharyngeal cancer has been developed and widely accepted by endoscopists in JapanVideo 1 A nerve-preserving strategy for endoscopic submucosal dissection of superficial pharyngeal cancers.ESD was performed under general anesthesia. Narrow-band imaging (NBI) and Lugol chromoendoscopy clearly revealed the lesion . We perEndoscopy_UCTN_Code_TTT_1AO_2AG


A 51-year-old man was referred to our institution with persistent iron deficiency anemia. Initial gastroduodenoscopy and colonoscopy at his local hospital were unremarkable. A subsequent small bowel capsule endoscopy revealed a distal small bowel polyp with evidence of fresh bleeding . A tripDuring intraoperative enteroscopy, an actively bleeding polyp was detected in the distal ileum , and a Video 1 The intraoperative enteroscopy procedure.Obscure gastrointestinal bleeding (OGIB) is a challenging condition that accounts for nearly 5 % of all gastrointestinal bleeding casesEndoscopy_UCTN_Code_CCL_1AC_2AC


These methods thus are hard to apply on real-world datasets (like ImageNet) since there are no such pre-defined attributes in the data environment. The latest works have explored to use semantic-rich knowledge graphs (such as WordNet) to substitute pre-defined attributes. However, these methods encounter a serious “role=“presentation”>domain shift” problem because such a knowledge graph cannot provide detailed enough semantics to describe fine-grained information. To this end, we propose a semantic-visual shared knowledge graph (SVKG) to enhance the detailed information for zero-shot learning. SVKG represents high-level information by using semantic embedding but describes fine-grained information by using visual features. These visual features can be directly extracted from real-world images to substitute pre-defined attributes. A multi-modals graph convolution network is also proposed to transfer SVKG into graph representations that can be used for downstream zero-shot learning tasks. Experimental results on the real-world datasets without pre-defined attributes demonstrate the effectiveness of our method and show the benefits of the proposed. Our method obtains a +2.8%, +0.5%, and +0.2% increase compared with the state-of-the-art in 2-hops, 3-hops, and all divisions relatively.Almost all existing zero-shot learning methods work only on benchmark datasets ( In recent years, zero-shot learning has attracted widespread attention in computer vision and machine learning areas. It aims to predict new classes that appear during the training process. The base idea of zero-shot learning is to use labeled semantic information  to learn a projection between semantic space and visual space. Then, visual samples from new classes can be projected into semantic space to match the attributes to decide the corresponding classifications. In the past five years, a large number of methods for zero-shot learning have been proposed based on this idea  .etc. Moreover, these datasets are quite small, only tens or hundreds classes and tens of thousands samples, which are far away from real-world data environment, normally tens of thousands classes and millions samples. Thus, existing zero-shot methods are hard to be applied in real-world data environments such as ImageNet which does not provide any pre-defined attributes for any classes.However, these zero-shot learning methods inevitably need to follow a major premise that semantic attributes should be pre-defined and labeled in the datasets. Almost all recent zero-shot learning works are only evaluated on six small benchmark datasets . etc.) to substitute pre-defined attributes in semantics and have better generalization power in semantic representation. However, these GCN-based zero-shot learning methods for image classification on the ImageNet dataset, like GCNZ (etc.) without fine-grained semantics . Thus, the core question becomes “can a knowledge graph provide fine-grained semantics or even more for zero-shot learning?”.In the past four years, the graph neural network (GNN) has been adopted by zero-shot learning, which makes zero-shot learning tasks applicable in real-world data environments . This isike GCNZ , DGP . At last, a multi-modals GCN network is proposed to embed (SVKG) into graph representations that can be used in zero-shot learning tasks, as showed in Based on the idea, we first scan all the image samples of seen classes in ImageNet-1K to extract parts of visual features for each seen class. Then, WordNet nodes and relations are extracted according to the seen and unseen class labels in ImageNet-1K and embedded as semantic features by a word embedding model. After that, parts of visual features and WordNet semantic features are connected together as a semantic-visual shared knowledge graph , then learn the semantic attribute characteristics of visual objects, and finally judge whether the attribute combinations are satisfied by the visual objects.Attribute-based methods account for the largest proportion of zero-shot learning (ZSL) research since ZSL was first proposed in DAP  in 2009 Following the design principles of AwA dataset, a series of benchmark datasets with pre-defined attributes are established today, including AwA2, CUB, SUN, FLO and aPY. Based on these benchmark datasets, ZSL has a blowout development with several milestones.In 2013, with the development of semantic embedding technology, the first milestone of ZSL was the ALE model proposed by In 2017, with the development of deep learning technology in the field of visual computing, the second milestone of ZSL is the SAE model proposed by In 2018, with the remarkable performance of Generative Adversarial Networks (GAN)  in imageRecently, some researchers have started to investigate how to mix various types of semantics  to address further “domain shift” issues. For instance, However, the above methods use pre-defined properties as their primary semantic source. As a result, these methods are essentially restricted to benchmark datasets with pre-defined features. The applicability of these methods in real-world environments devoid of pre-defined features is restricted to a small number of cases.Knowledge graph (KG) actually is a third part knowledge base that can provide semantic information for semantic-to-visual transformation in zero-shot learning. Knowledge graph (KG) actually constitutes a third-party knowledge base that can provide semantic information for semantic-to-visual transformation in zero-shot learning. Thus, KG is intuitively considered a substitution for pre-defined attributes. Benefiting from the development of GNN, GCNZ  creates However, compared to the pre-defined features in the six benchmark datasets, the knowledge network used in GNN-based approaches  providesTo enhance the fine-grained semantic information for existing KG, the latest research tries to apply “visual knowledge” in zero-shot learning. However, the above methods continue to see visual semantics as auxiliary or augmented semantics. In contrast to them, we want to combine semantic and visual representations into a single common knowledge graph, so-called visual-semantic shared knowledge graph.X ∈ ℝN×F be the word-embedding set for all the nodes in A ∈ ℝN×N be the adjacency matrix transferred from F is the dimensions of the embedding. After, a graph representation model gΘ is learned to transfer X to the graph node representation H ∈ ℝN×F′ supervised by Iseen. Here, Iseen ∈ ℝN×F′′ is the seen image feature set extracted by a CNN model. At last, an image classification model Iunseen to the particular class according to the graph node representation H. Here, Iunseen ∈ ℝN×F′′ is the unseen image feature set extracted by a CNN model.Knowledge graph-based Zero-Shot learning uses the knowledge-aided method to learn an image classification model from seen classes to predict unseen classes. First, given a knowledge graph SVKG) is a multi-modal graph that contains both semantic embedding and visual embedding in the same graph. Let X represents the embedding set of the graph nodes with Xs denotes Word-Embedding nodes and Xv represents CNN-Embedding nodes. A represents the edge set while As denotes the edges among Word-Embedding nodes and Av denotes the edges among CNN-Embedding nodes. SVKG is defined in Algorithm 1.Semantic-visual shared knowledge graph , seen images Output: SV KG  1:  Xs ←− Put X into Glove  2:  As ←− Put X into WordNet hyponym/hypernym  3:  Xv ←− Put seen images into EfficientDet  4:  As ←− Put Xv connecte with semantic object node  5:  X′ ←− Xs ∪ Xv  6:  A′ ←− As ∪ Av  7:  SV KG ←−{ X′, A′ }  8:  return  SV KG  _______________________________________________________________________________    SVKG, the semantic node set Xs is constructed from WordNet (https://wordnet.princeton.edu/) Noun words and embedded into word features by Glove (https://nlp.stanford.edu/projects/glove/). The edge matrix As is established by WordNet hyponym/hypernym links among these words. The visual node set Xv is obtained by using EfficientDet (etc.) from ImageNet-1k. Then, these visual nodes are connected to their semantic object node by the edge matrix Av . An example of SVKG about bird Finch is presented in In cientDet  to detecXs needs to be transferred into semantic-visual shared space H. However, the visual feature space  Xv might cause interference during Xs⟶H transfer process, leading to a serious over-fitting problem. Empirically, many data augmentation methods have shown to improve the generalization and robustness of the learning model . In SVKGaug, there are two feature modals. One is semantic feature modal {Xs generated from Glove-Embedding. The other is visual feature modal Xv extracted by EfficientDet. Thus, multi-modals graph representation learning is aimed to transfer both Xs and Xv into a semantic-visual shared feature modal H.As explained in the ‘Problem Definition’ section, knowledge graph Xs and Xv into H, as defined in gΘ is a Graph Convolutional Network (GCN) employed from D is the diagonal degree matrix of A matrix, where Di,i). Then, Iunseen is input into the trained classification model to calculate the similarities with all graph features H′. To obtain the predicting result, Resultset records all the similarities between H′ and Iunseen, and the labeli of corresponding In the predicting process, we first use the same Resnet-50 model to extract unseen image feature  ________________________________________________________________________________________ Algorithm 2 Zero-shot Predicting Process _______________________________________________________________________________________ Input: H′, Iunseen Output: Top-k result  1: Resultset ←−{∅, ∅}  2: for H′i in H′ do  3:   Simi ←− H′i ⋅ Iunseen  4:   lableli ←− findLabel(H′i)  5:   Resultset ←−{Simi, lableli}  6: end for  7: Resultset ←− SortBySim(Resultset)  8: Resultset ←− Top-k(Resultset)  9: return  Resultset  ____________________________________________________________________________________    Iunseen is not used during the whole training process. In the predicting process, the input Iunseen is the first time the model touches the unseen images.It can be observed, the unseen image feature SVKG. At last, extra downstream tasks and observations are discussed.In this section, the datasets and evaluation metrics are introduced first. Then, the implementation details about the model settings are explained. After, the state-of-the-arts comparisons are presented. Then, an ablation study is conducted for the proposed nth jumps to connect to other classes of ImageNet through the relations of a WordNet graph. For example, as showed in The experiments are conducted on ImageNet , which iXv in SVKG, a “Detailed” group which contains four subsets  are divided from ImageNet-1K and corresponding parts visual features Xv for all these detailed categories are extracted from their image samples by EfficientDet. In a short, the experiments involve two groups and seven divisions divided by ImageNet. The detailed information for each corresponding division is shown in In order to show the effectiveness of visual feature There, the label rate demonstrates the difficulty of the corresponding division. This is because with the reduction of supervision in semi-supervised manner, there are higher requirements for the generalization and robustness of the model. The performance of corresponding division will also degrade as the supervised label rate decreases. Therefore, the accuracy in widely different divisions can reflect the robustness of the model. As can be seen from The proposed method is evaluated according to Generalized Zero-Shot Learning (GZSL) setting, which is the most challenging evaluation metric for zero-shot learning. In GZSL, all classes, no matter seen or unseen classes, are all considered as the candidate classes in the testing, while the test samples are only from unseen classes. We adopt the same train/test splits and the Top-k Hit Ratio (Hit@k) metric, in accordance with In the model implementations, Resnet-50  is used In this section, all the comparisons conducted in both General and Detailed groups follow the GZSL setting. For General group, DeViSE , ConSE , ConSE2 General group comparisons: From But from Top-2 to Top-20 accuracy evaluations, the latest method CL-DKG surpasses our method. The reason might be the fine-grained features strengthen discrimination ability but weaken, to some extent, the generalization ability of the model. However, it can be seen from Detailed group comparisons: The most significant contribution of the proposed method is that fine-grained visual features are shared with semantic features in the same knowledge graph. Here, Detail comparisons are aimed to show the effects of such shared visual features through some representative categories of ImageNet. It can be seen from From SVKG mainly consists of Xs, Xv and Xaug. Thus, four combinations, “Xs” only, “Xs+ Xv”, “Xs+ Xaug” and “All”, for ablation study are established. Indeed, “Xs” denotes the traditional knowledge graph while “All” represents the proposed knowledge graph SVKGaug. The experiments are conducted also on the Detailed group  to 17.7% (All). The reason is that dog is a well-known category and widely used everywhere. This leads the upstream feature extractor, EfficientDet, to work well and extract more accurate part visual features for SVKG that significantly improves the Top-1 accuracy. A similar phenomenon is also observed in bird and primate categories. The study thus also reveals that the better performance of the upstream feature extractor, the better accuracy the proposed SVKG might achieve. In a short, the study demonstrates the effectiveness of each proposed module in real-world zero-shot learning tasks.As defined in ed group  to evaluXs+ Xaug”. The influence of graph augmentation Xaug is weakened after two or three hops, and distant nodes are easy to overfit with the presence of part visual nodes. Therefore, in the long-distance prediction, the overall performance of the model decreases.In the cases from Top-5 to Top-20, the best performance is “ For further discussion, we conducted an unseen class search compared with DGP on Detailed group. Some representative results are presented in SVKG graph on dog category. The orange-colored nodes represent seen classes and the blue-colored nodes represent unseen classes. SVKG. Intuitively, SVKG is more structured and order than the graph features of GCNZ and DGP. It can be observed that blue-colored nodes distribute among orange-colored nodes reveals the hidden relationships between seen and unseen classes. This is because the proposed visual features Xv lead the graph representations of SVKG close to real-world dog taxonomy. Unseen class nodes thus are distributed to the most related seen class nodes by the meaning of taxonomy. It obviously alleviates coincidence and proximity during unseen class predicting that improves the performance of zero-shot learning.Meanwhile, t-SNE was used to visualize knowledge graph features for initial graph, GCNZ graph, DGP graph and SVKG with GCNZ graph feature distribution and real image feature distribution, as shown in SVKG is significantly closer to the real image feature distribution than GCNZ. This indicates the feature of SVKG is more close to the real visual feature that is easier to be matched with unseen classes. This also explains why the proposed method achieves the best accuracy on Top-1 and Top-2 unseen class predicting.Moreover, we also compared feature distribution of SVKG) for zero-shot learning on the real-world dataset without needing pre-defined attributes. It combines semantic (from WordNet and Glove embedding) and visual features (extracted from raw images by EfficientDet) together in the same graph. The visual feature provides detailed information for describing fine-grained semantics that alleviates the “Domain-Shift” problem during the semantic-to-visual transformation of zero-shot learning. A novel multi-modal GCN model is also proposed to learn the graph representations of SVKG. After, the graph representations are further used for downstream zero-shot learning tasks in the experiments.In this article, we propose a semantic-visual shared knowledge graph (Experimental results on the real-world dataset demonstrate the effectiveness of our method and illustrate the multi-modal graph guide model generates more discriminative representation. And our method significantly surpasses other methods on Top-1 to Top-5 accuracy evaluations in the bird, snake, primate, and dog categories. Especially, on Top-1 accuracy evaluations, our method even achieves a 2-3 times increase compared with the state-of-the-art.SVKG is only storing fine-grained information in parts of visual features. In the future, we will add color, material, shape, and other relations and associated nodes to the SVKG in order to further increase the model’s performance.The important component of zero-shot learning tasks implemented in real-world environments is still how to reasonably use and construct the knowledge graph. In this paper, the 10.7717/peerj-cs.1260/supp-1Supplemental Information 1Click here for additional data file.10.7717/peerj-cs.1260/supp-2Supplemental Information 2Click here for additional data file.


The term “RNA-seq” refers to a collection of assays based on sequencing experiments that involve quantifying RNA species from bulk tissue, from single cells, or from single nuclei. The kallisto, bustools, and kb-python programs are free, open-source software tools for performing this analysis that together can produce gene expression quantification from raw sequencing reads. The quantifications can be individualized for multiple cells, multiple samples, or both. Additionally, these tools allow gene expression values to be classified as originating from nascent RNA species or mature RNA species, making this workflow amenable to both cell-based and nucleus-based assays. This protocol describes in detail how to use kallisto and bustools in conjunction with a wrapper, kb-python, to preprocess RNA-seq data. While multiple steps are necessary to process input consisting of FASTQ sequencing files, a reference genome FASTA, and a GTF annotation28, to an output of quantifications using kallisto and bustools, these steps are greatly facilitated by the wrapper tool, kb-python. kb-python can extract reference transcriptomes from reference genomes and run kallisto and bustools in workflows optimal for each assay type. The kb-python tool simplifies the running of kallisto and bustools to the extent that all of this can be done in two steps: `kb ref` for generating a kallisto index from an annotated reference genome and `kb count` for mapping and quantification. Thus, kallisto, bustools, and kb-python make RNA-seq preprocessing efficient, modular, flexible, and simple.1The preprocessingindex from a set of sequences, referred to as targets, representing the set of sequences that RNA-seq reads can be mapped to. In a standard analysis, these targets are usually transcript sequences . However, more generally, users can define targets from any sets of sequences they wish to map their sequencing reads against. Since kallisto is a tool that leverages pseudoalignment, the mapping procedure relies on read assignment, such that each read is deemed to be compatible with a certain set of targets, rather than standard alignment. The kallisto index is based on the Bifrost29 implementation of the colored de Bruijn graph30, which enables memory-efficient and rapid read assignment.For RNA-seq read mapping, kallisto builds an kb ref command . The index created by --workflow=nac (nac: nascent and cDNA) contains both the cDNA and the nascent transcripts. The nascent transcript sequences consist of the full gene (both exons and introns). This nac index is suitable for single-nucleus RNA-seq as there exists a high abundance of non-mature transcripts captured in nucleus-based sequencing assays.32 Additionally, this nac index should be used for analyses that require jointly modeling nascent and mature RNA species.38 For both the standard and nac workflows, a user supplies a genome FASTA and GTF annotation, which kb-python uses to extract the relevant sequences. Finally, if one wishes to index a custom set of targets or of k-mers  allows specific biotypes to be selected from the GTF file, making possible filtering of entries such as pseudogenes, which can improve read mapping accuracyry usage . It is rkb count command within kb-python enables mapping and quantification of bulk, single-cell, and single-nucleus RNA-seq reads (The eq reads . As diffThe specifications for sequencing assay technology within kb-python are as follows:Technology string: A technology string for a particular type of assay can be supplied via the -x option. The technology string can be used in one of three ways:kb --list) so one can name one of those directly (e.g. one can specify -x 10xv3).Option 1: Several assays are predefined within the software , and the biological sequence that is to be mapped .Strandedness: If a read (or the first read in the case of paired-end reads) is to be mapped in forward orientation, one should specify --strand=forward. If it is to be mapped in reverse orientation, one should specify --strand=reverse. If one does not want to map reads with strand-specificity, then one should specify --strand=unstranded. If a predefined name is used in the technology string -x option (option 1), then kb-python uses a default stranded option for that technology ; otherwise, the default is unstranded. Setting the --strand option explicitly will overrule the default option.Parity: If the technology involves two biological read files that are derived from paired-end sequencing (as is the case with Smartseq243 and Smartseq344 and many bulk RNA sequencing kits), one should specify --parity=paired to perform mapping that takes into account the fact that the reads are paired-end. Otherwise, one can specify --parity=single. If a predefined name is used in the -x technology string option (option 1), then kb-python uses the default parity option for that technology .On list: For single-cell and single-nucleus sequencing assays, barcodes are used to identify each cell or nucleus. The “on list” of barcodes represents the known barcode sequences that are included in the assay. Barcodes extracted from the sequencing reads will be error-tolerantly mapped to this list in a process known as barcode error correction. The on list filename can be specified with the -w option in kb count. It can also be obtained by seqspec.40 If an on list is not provided or cannot be found for the given technology, then an on list is created by bustools via the bustools allowlist command which identifies repeating barcodes in sequencing reads. If the technology does not include cell barcodes (as is the case in bulk RNA-seq), the “on list” option is irrelevant and no barcode processing occurs which should be the case for assays that don’t include cell/nuclei barcodes . If a predefined name is used in the -x technology string option (option 1), then kb-python uses the default on list option for that technology.--workflow=nac should be used in kb count so that the nascent and mature RNA species are quantified accurately; otherwise that option should be omitted or --workflow=standard (which is the default) can be explicitly specified. For the nac workflow, one obtains three count matrices: 1) nascent, 2) mature, and 3) ambiguous. In most experiments, the plurality of reads will be “ambiguous” since they originate from exons, which are present in both nascent RNA and mature RNA. Therefore, it is desirable to generate additional matrices by adding the counts from those three matrices, which users can either do themselves or by using the --sum option.24--sum=total adds all three matrices, --sum=cell adds the mature and ambiguous matrices, and --sum=nucleus adds the nascent and ambiguous matrices. Different matrices may be used for different types of analyses. For example, in single-cell RNA-seq analysis (where most “ambiguous” counts are likely of mature RNA origin), jointly modeling the mature+ambiguous count matrix (--sum=cell) with the nascent count matrix permits biophysical modeling of RNA processing.38 In single-nucleus RNA-seq quantification, one might want to use --sum=nucleus to add up the nascent+ambiguous counts. The kb-python, kallisto, and bustools commands for the standard workflow and the nac workflow are shown in If a nac index was generated by kb ref, 48, sleuth49, limma-voom51, and other differential gene expression programs.In addition to single-cell and single-nucleus RNA-seq, kb count can be used for bulk RNA-seq. Bulk RNA-seq generally does not have UMIs or cell barcodes  and relies on cDNA mapping. With -x BULK as the technology string, a workflow specific for bulk RNA-seq quantification is executed . This wiTo facilitate multi-sample analysis, artificial unique sample-specific barcodes can be created and stored in the BUS file and the resulting mapping between the artificially generated barcode and the sample ID is outputted. These sample-specific barcodes are 16-bp in length and are also stored in the BUS file. Where there exists both a cell barcode (like in single-cell RNA-seq) and a sample-specific barcode, both sets of barcodes will be outputted so that each entry in the resulting output count matrix can be associated with a particular cell and a particular sample. To utilize the multi-sample workflow, a batch file containing the file names of the FASTQ files must be provided .--dry-run option in kb count outputs the kallisto and bustools commands that will be run without actually running the programs. Also, the --verbose option in kb count is helpful for examining the kallisto and bustools commands that are being run as well as their output.The technical details of how kb count utilizes kallisto and bustools are detailed in the following paragraph. Note that the kallisto bus command within kallisto to produce a BUS file, which stores the read mapping information, and then uses bustools23 commands to process the BUS file. The kallisto bus command maps RNA-seq reads to a kallisto index, and the resultant BUS file stores the mapping information, including the barcode, unique molecular identifier (UMI), and the equivalence class representing the set of transcripts the read is compatible with.23 In certain RNA-seq assays, barcodes and/or UMIs may not be present, and are therefore not considered when processing the BUS file. After the mapping step is complete, the BUS file is sorted via the bustools sort command to facilitate further processing. For single-cell and single-nucleus experiments with multiplexed barcodes in the sequencing reads, an “on list” of barcodes, representing the known barcode sequences that are included in the assay, needs to be provided. If an “on list” is unavailable, the bustools allowlist command can be used to construct one from a sorted BUS file. The barcodes in the sorted BUS file are error-corrected to the “on list” via bustools correct, then the BUS file is sorted again with bustools sort. The final sorted, on list-corrected BUS file is then used to generate quantifications via count matrices through the bustools count command. At any point, a sorted BUS file can be inputted into bustools compress to create a compressed BUS file (a BUSZ file), which can be subsequently decompressed via bustools decompress.52 Other bustools features enable more specialized workflows beyond what is provided by kb-python , one can supply the --cm option for quantification.Gene-level count matrices: In single-cell and single-nucleus RNA-seq, typically a gene-level count matrix is produced by collapsing UMIs to the gene level. Here, the bustools count command is run with the --genecounts option is not provided, and --multimappingis provided to avoid discarding reads or collapsed UMIs that are assigned to multiple genes. If UMIs are not present in the sequencing technology, the --cm option is supplied to perform counting without UMI collapsing. While downstream analyses can be performed on TCCs55, it is more often useful to produce transcript-level abundances from the TCCs for technologies where sequencing reads span the full length of transcripts, such as bulk RNA-seq. In such cases, an expectation-maximization algorithm is typically performed to probabilistically estimate transcript abundances.56 The procedure to generate transcript-level abundance matrices is performed by running the kallisto quant-tcc command on the TCC matrices.Transcript-level count matrices: Transcript-compatibility counts (TCCs) are counts assigned to equivalence classes where each equivalence class is defined by a unique set of transcripts. For producing a matrix of transcript-compatibility counts (TCCs), the 40 provides a specification for the structure of genomic sequencing assays, formatted in a machine-readable YAML file. The specification can be readily inputted into the kallisto bustools workflow for preprocessing reads from a given assay  being the matrix columns.Here, the quantification output of the kb count command is described. While the initial step of kb count uses kallisto to produce a BUS file located at output_dir/output.bus, the actual quantification results are located in matrices in subdirectories of output_dir/. All matrices have the extension .mtx and will be in a sparse matrix (Matrix Market) file format with the barcodes (i.e. the cells or samples) being the matrix rows and the genes :The standard workflowcells_x_genes.mtx: The count matrix (in Matrix Market file format); only exonic reads are countedcells_x_genes.barcodes.txt: The barcodes representing the matrix row namescells_x_genes.genes.txt: The gene IDs representing the matrix column namescells_x_genes.genes.names.txt: Same as cells_x_genes.mtx except with gene names instead of gene IDs for the matrix columnscells_x_genes.barcodes.prefix.txt: If sample-specific barcodes are generated in addition to cell barcodes being recorded, then this file will be created and the sample-specific barcodes will be stored here. The lines of this file correspond to the lines in the cells_x_genes.barcodes.txt which contains the cell barcodes (both files will have the same number of lines). The sample-specific barcodes and cell barcodes can be joined together as a unique identifier for downstream analysis.nac workflow: same as the standard workflow except the .mtx files produced are differentcells_x_genes.mature.mtx: The mature RNA count matrixcells_x_genes.ambiguous.mtx: The nascent RNA count matrixcells_x_genes.nascent.mtx: The ambiguous RNA count matrixcells_x_genes.cell.mtx: The mature+ambiguous RNA count matrix (note: this is what is quantified into the count matrix in the standard workflow)cells_x_genes.nucleus.mtx: The nascent+ambiguous RNA count matrixcells_x_genes.total.mtx: The mature+nascent+ambiguous RNA count matrix--tcc option is used.For RNA-seq assays (e.g. bulk RNA-seq or Smartseq2) that profile the full length of transcripts in which case it is desirable to perform transcript-level quantification, the output_dir/counts_unfiltered/ which contains the following files for the standard workflow:The first step to doing transcript-level quantification is to obtain transcript-compatibility counts (TCCs) over equivalence classes (ECs). The TCCs will be outputted into cells_x_tcc.mtx: The count matrix containing the TCCscells_x_tcc.barcodes.txt: The barcodes representing the matrix row namescells_x_tcc.ec.txt: The equivalence classes representing the matrix column names  for all transcripts within the equivalence class)output_dir/quant_unfiltered/ directory which contains the following:The --tcc option will additionally produce transcript-level estimated counts which will be placed in the matrix.abundance.mtx: The matrix containing the transcript-level estimated countsmatrix.abundance.tpm.mtx: The matrix containing the TPM-normalized transcript-level abundancesmatrix.efflens.mtx: A matrix that contains the transcript effective lengthsmatrix.fld.tsv: A file with two columns, containing the mean and standard deviation, respectively, of the fragment length distribution used to produce transcript-level abundances and effective lengths for each row of the matrix.matrix.abundance.gene.mtx: A matrix that is the same as the matrix.abundance.mtx matrix except counts are aggregated to gene-levelmatrix.abundance.gene.tpm.mtx: A matrix that is the same as the matrix.abundance.tpm.mtx matrix except TPMs are aggregated to gene-leveltranscripts.txt: The transcript names representing the matrix column names for the transcript-level quantification matricesgenes.txt: The gene IDs representing the matrix column names for the gene-level aggregation quantification matricestranscript_lengths.txt: The transcript names along with their lengthsrow names are the individual samples and will be the same as those in output_dir/counts_unfiltered/cells_x_tcc.barcodes.txt - The output_dir/matrix.cells and output_dir/matrix.sample.barcodes files provide a mapping between the sample name and the sample barcode.*Note: The *Note: The --matrix-to-directories option will output each row of the matrix into a separate subdirectory. In other words, using this option will produce multiple new directories within output_dir/quant_unfiltered/. Each one will be named abundance_{n} . Within each subdirectory, an abundance.tsv text file and abundance.h5 HDF5 file will be created containing the quantifications for that particular sample. These abundance files are identical to the abundance files produced by the original version of kallisto for bulk RNA-seq.60 for downstream processing in python, an anndata61 object needs to be created , the matrices that are loaded in will become much smaller and more efficient to process.A 64-bit computer running either macOS, Windows, or a Linux/Unix operating system.kallisto version 0.50.1 or later (which comes packaged with kb-python)bustools version 0.43.1 or later (which comes packaged with kb-python)kb-python version 0.28.0 or laterPython 3.7 or later (for kb-python version 0.28.0)Bulk, single-cell, or single-nucleus RNA sequencing reads in (possibly gzip) FASTQ format.unfiltered GTF file) and an hour to generate the nac index. For the preprocessing of 800 million Illumina sequencing reads (stored in a single pair of fastq.gz files) produced by single-cell RNA-seq from 10x Genomics, kb count with the nac workflow can take under an hour on 8 threads and under 40 minutes on 16 threads, with an even lower runtime for the standard workflow.The runtime depends on the size of the reference being indexed, the number and length of the sequencing reads being processed, other properties of the dataset being quantified, system hardware, and the number of threads allotted. The kb ref command only needs to be run once to create the index against which reads will be mapped. With 8 threads on a server with x86-64 architecture and 32 Intel Xeon CPUs (E5-2667 v3 @ 3.20GHz), kb ref takes approximately 15 minutes to generate a standard index from the GRCm39 mouse genome  file created by kb ref should be the exact file used by kb count when running kb count on that index. All the transcripts in the t2g file must be exactly the same as the transcripts present in the kallisto index. Incompatibilities can lead to unpredictable behavior in the bustools quantification step.31 .dna.toplevel.fa.gz files or the GENCODE67 .primary_assembly.genome.fa.gz files should be used as the reference genome. Use of FASTA files incompatible with the supplied GTF will lead to errors.When using kb ref to generate a kallisto index, a genome FASTA file (not a transcriptome FASTA file) should be supplied along with the genome annotation GTF file. A transcriptome file will automatically be generated by kb ref and be indexed by kallisto. In general, the EnsemblWhen performing multiple kb-python runs simultaneously, a different temporary directory must be specified via the --tmp option for each run. The temporary directory also must not exist beforehand.Finally, one should make sure that the value supplied to the -x technology string option matches the assay from which the sequencing reads were generated.Note: If the technology string begins with a −, for example: −1,0,0:0,0,5:0,5,0, one would need to write −x “ −1,0,0:0,0,5:0,5,0” to avoid the string being misinterpreted as a command-line flag.Here, we describe the procedures to use for mouse samples of paired-end bulk RNA-seq, 10x (version 3) single-cell RNA-seq, and 10x (version 3) single-nucleus RNA-seq.Input:Paired-end unstranded mouse RNA-seq reads (3 samples):sample1_R1.fastq.gz sample1_R2.fastq.gzsample2_R1.fastq.gz sample2_R2.fastq.gzsample3_R1.fastq.gz sample3_R2.fastq.gz1.pip install kb_python2.ftp.ensembl.org/pub/release-108/fasta/mus_musculus/dna/Mus_musculus.GRCm39.dna.primary_assembly.fa.gzwget ftp.ensembl.org/pub/release-108/gtf/mus_musculus/Mus_musculus.GRCm39.108.gtf.gzwget 3.kb ref -i index.idx -g t2g.txt -f1 cdna.fasta \    Mus_musculus.GRCm39.dna.primary_assembly.fa.gz \    Mus_musculus.GRCm39.108.gtf.gz4.kb count -x BULK -o output_dir -i index.idx -g t2g.txt \    --parity=paired --strand=unstranded \    --tcc --matrix-to-directories \    sample1_R1.fastq.gz sample1_R2.fastq.gz \    sample2_R1.fastq.gz sample2_R2.fastq.gz \    sample3_R1.fastq.gz sample3_R2.fastq.gz5.Output for sample 1:output_dir/quant_unfiltered/abundance_1/abundance.tsvoutput_dir/quant_unfiltered/abundance_1/abundance.gene.tsvoutput_dir/quant_unfiltered/abundance_1/abundance.h5Output for sample 2:output_dir/quant_unfiltered/abundance_2/abundance.tsvoutput_dir/quant_unfiltered/abundance_2/abundance.gene.tsvoutput_dir/quant_unfiltered/abundance_2/abundance.h5Output for sample 3:output_dir/quant_unfiltered/abundance_3/abundance.tsvoutput_dir/quant_unfiltered/abundance_3/abundance.gene.tsvoutput_dir/quant_unfiltered/abundance_3/abundance.h5The abundance.tsv files contain the transcript-level abundances. The abundance.h5 file contains the same information as the abundance.tsv files except in HDF5 format. The abundance.gene.tsv files contain the gene-level abundances (taken by summing up the transcript-level abundances for each gene). These files can be used in downstream differential gene expression programs.Input:10x version 3 single-cell RNA-seq reads: R1.fastq.gz and R2.fastq.gz1.pip install kb_python2.ftp.ensembl.org/pub/release-108/fasta/mus_musculus/dna/Mus_musculus.GRCm39.dna.primary_assembly.fa.gzwget ftp.ensembl.org/pub/release-108/gtf/mus_musculus/Mus_musculus.GRCm39.108.gtf.gzwget 3.kb ref -i index.idx -g t2g.txt -f1 cdna.fasta \    Mus_musculus.GRCm39.dna.primary_assembly.fa.gz \    Mus_musculus.GRCm39.108.gtf.gz4.kb count -x 10xv3 -o output_dir -i index.idx -g t2g.txt \    R1.fastq.gz R2.fastq.gz5.Output:output_dir/counts_unfiltered/cells_x_genes.mtxoutput_dir/counts_unfiltered/cells_x_genes.barcodes.txtoutput_dir/counts_unfiltered/cells_x_genes.genes.txtoutput_dir/counts_unfiltered/cells_x_genes.genes.names.txtThe cells_x_genes.mtx is the count matrix file with the barcodes (the row names) listed in cells_x_genes.barcodes.txt and the gene names (the column names) listed in cells_x_genes.genes.names.txt .Input:10x version 3 single-nucleus RNA-seq reads: R1.fastq.gz and R2.fastq.gz1.pip install kb_python2.ftp.ensembl.org/pub/release-108/fasta/mus_musculus/dna/Mus_musculus.GRCm39.dna.primary_assembly.fa.gzwget ftp.ensembl.org/pub/release-108/gtf/mus_musculus/Mus_musculus.GRCm39.108.gtf.gzwget 3.kb ref --workflow=nac -i index.idx -g t2g.txt \    -c1 cdna.txt -c2 nascent.txt -f1 cdna.fasta -f2 nascent.fasta \    Mus_musculus.GRCm39.dna.primary_assembly.fa.gz \    Mus_musculus.GRCm39.108.gtf.gz4.kb count -x 10xv3 --workflow=nac -o output_dir \    -i index.idx -g t2g.txt -c1 cdna.txt -c2 nascent.txt \    --sum=total R1.fastq.gz R2.fastq.gz5.Output:output_dir/counts_unfiltered/cells_x_genes.mature.mtxoutput_dir/counts_unfiltered/cells_x_genes.nascent.mtxoutput_dir/counts_unfiltered/cells_x_genes.ambiguous.mtxoutput_dir/counts_unfiltered/cells_x_genes.cell.mtxoutput_dir/counts_unfiltered/cells_x_genes.nucleus.mtxoutput_dir/counts_unfiltered/cells_x_genes.total.mtxoutput_dir/counts_unfiltered/cells_x_genes.barcodes.txtoutput_dir/counts_unfiltered/cells_x_genes.genes.txtoutput_dir/counts_unfiltered/cells_x_genes.genes.names.txtThis workflow can be used for both single-cell RNA-seq and single-nucleus RNA-seq. Many count matrix files (.mtx files) are generated. For quantification of total RNA present in each cell or nucleus, one would want to use the cells_x_genes.total.mtx. For biophysical models that jointly consider spliced and unspliced transcripts, one may want to use cells_x_genes.cell.mtx (for the “spliced” transcripts) and cells_x_genes.nascent.mtx (for the “unspliced” transcripts).The barcodes (the matrix row names) are listed in cells_x_genes.barcodes.txt and the gene names (the matrix column names) are listed in cells_x_genes.genes.names.txt .There are many ways to extend the standard workflows beyond bulk RNA-seq, 10x single-cell RNA-seq, and 10x single-nucleus RNA-seq. For an additional, extended example that involves preprocessing mouse multiplexed single-nucleus SPLiT-seq RNA-seq data with a filtered mouse genome annotation, see Supplement 1