ovrawm/t_starfysh.txt

#!/usr/bin/env python
# coding: utf-8

# # Deconvolution spatial transcriptomic without scRNA-seq
# 
# This is a tutorial on an example real Spatial Transcriptomics (ST) data (CID44971_TNBC) from Wu et al., 2021. Raw tutorial could be found in https://starfysh.readthedocs.io/en/latest/notebooks/Starfysh_tutorial_real.html
# 
# 
# Starfysh performs cell-type deconvolution followed by various downstream analyses to discover spatial interactions in tumor microenvironment. Specifically, Starfysh looks for anchor spots (presumably with the highest compositions of one given cell type) informed by user-provided gene signatures ([see example](https://drive.google.com/file/d/1AXWQy_mwzFEKNjAdrJjXuegB3onxJoOM/view?usp=share_link)) as priors to guide the deconvolution inference, which further enables downstream analyses such as sample integration, spatial hub characterization, cell-cell interactions, etc. This tutorial focuses on the deconvolution task. Overall, Starfysh provides the following options:
# 
# At omicverse, we have made the following improvements:
# - Easier visualization, you can use omicverse unified visualization for scientific mapping
# - Reduce installation dependency errors, we optimized the automatic selection of different packages, you don't need to install too many extra packages and cause conflicts.
# 
# **Base feature**:
# 
# - Spot-level deconvolution with expected cell types and corresponding annotated signature gene sets (default)
# 
# **Optional**:
# 
# - Archetypal Analysis (AA):
# 
#     *If gene signatures are not provided*
#     
#     - Unsupervised cell type annotation
# 
#     *If gene signatures are provided but require refinement*:
#     
#     - Novel cell type / cell state discovery (complementary to known cell types from the *signatures*)
#     - Refine known marker genes by appending archetype-specific differentially expressed genes, and update anchor spots accordingly
#     
# - Product-of-Experts (PoE) integration
# 
#     Multi-modal integrative predictions with expression & histology image by leverging additional side information (e.g. cell density) from H&E image.
# 
# He, S., Jin, Y., Nazaret, A. et al.
# Starfysh integrates spatial transcriptomic and histologic data to reveal heterogeneous tumor–immune hubs.
# Nat Biotechnol (2024).
# https://doi.org/10.1038/s41587-024-02173-8

# In[1]:


import scanpy as sc
import omicverse as ov
ov.plot_set()


# In[2]:


from omicverse.externel.starfysh import (AA, utils, plot_utils, post_analysis)
from omicverse.externel.starfysh import _starfysh as sf_model


# ### (1). load data and marker genes
# 
# File Input:
# - Spatial transcriptomics
#     - Count matrix: `adata`
#     - (Optional): Paired histology & spot coordinates: `img`, `map_info`
# 
# - Annotated signatures (marker genes) for potential cell types: `gene_sig`
# 
# Starfysh is built upon scanpy and Anndata. The common ST/Visium data sample folder consists a expression count file (usually `filtered_featyur_bc_matrix.h5`), and a subdirectory with corresponding H&E image and spatial information, as provided by Visium platform.
# 
# For example, our example real ST data has the following structure:
# ```
# ├── data_folder
#     signature.csv
# 
#     ├── CID44971:
#         \__ filtered_feature_bc_mactrix.h5
# 
#         ├── spatial:
#             \__ aligned_fiducials.jpg
#                 detected_tissue_image.jpg
#                 scalefactors_json.json
#                 tissue_hires_image.png
#                 tissue_lowres_image.png
#                 tissue_positions_list.csv
# ```
# 
# For data that doesn't follow the common visium data structure (e.g. missing `filtered_featyur_bc_matrix.h5` or the given `.h5ad` count matrix file lacks spatial metadata), please construct the data as Anndata synthesizing information as the example simulated data shows:

# [Note]: If you’re running this tutorial locally, please download the sample [data](https://drive.google.com/drive/folders/1RIp0Z2eF1m8Ortx0sgB4z5g5ISsRFzJ4?usp=share_link) and [signature gene sets](https://drive.google.com/file/d/1AXWQy_mwzFEKNjAdrJjXuegB3onxJoOM/view?usp=share_link), and save it in the relative path `data/star_data` (otherwise please modify the data_path defined in the cell below):

# In[3]:


# Specify data paths
data_path = 'data/star_data'
sample_id = 'CID44971_TNBC'
sig_name = 'bc_signatures_version_1013.csv'


# In[4]:


# Load expression counts and signature gene sets
adata, adata_normed = utils.load_adata(data_folder=data_path,
                                       sample_id=sample_id, # sample id
                                       n_genes=2000  # number of highly variable genes to keep
                                       )


# In[5]:


import pandas as pd
import os
gene_sig = pd.read_csv(os.path.join(data_path, sig_name))
gene_sig = utils.filter_gene_sig(gene_sig, adata.to_df())
gene_sig.head()


# **If there's no input signature gene sets, Starfysh defines "archetypal marker genes" as *signatures*. Please refer to the following code snippet and see details in section (3).**
# 
# ```Python
# aa_model = AA.ArchetypalAnalysis(adata_orig=adata_normed)
# archetype, arche_dict, major_idx, evs = aa_model.compute_archetypes(r=40)
# gene_sig = aa_model.find_markers(n_markers=30, display=False)
# gene_sig = utils.filter_gene_sig(gene_sig, adata.to_df())
# gene_sig.head()
# ```

# In[6]:


# Load spatial information
img_metadata = utils.preprocess_img(data_path,
                                    sample_id,
                                    adata_index=adata.obs.index,
                                    #hchannel=False
                                    )
img, map_info, scalefactor = img_metadata['img'], img_metadata['map_info'], img_metadata['scalefactor']
umap_df = utils.get_umap(adata, display=True)


# In[7]:


import matplotlib.pyplot as plt
plt.figure(figsize=(6, 6), dpi=80)
plt.imshow(img)


# In[8]:


map_info.head()


# ### (2). Preprocessing (finding anchor spots)
# - Identify anchor spot locations.
# 
# Instantiate parameters for Starfysh model training:
# - Raw & normalized counts after taking highly variable genes
# - filtered signature genes
# - library size & spatial smoothed library size (log-transformed)
# - Anchor spot indices (`anchors_df`) for each cell type & their signature means (`sig_means`)
# 

# In[ ]:


# Parameters for training
visium_args = utils.VisiumArguments(adata,
                                    adata_normed,
                                    gene_sig,
                                    img_metadata,
                                    n_anchors=60,
                                    window_size=3,
                                    sample_id=sample_id
                                   )

adata, adata_normed = visium_args.get_adata()
anchors_df = visium_args.get_anchors()


# In[10]:


adata.obs['log library size']=visium_args.log_lib
adata.obs['windowed log library size']=visium_args.win_loglib


# In[11]:


sc.pl.spatial(adata, cmap='magma',
                  # show first 8 cell types
                  color='log library size',
                  ncols=4, size=1.3,
                  img_key='hires',
                  #palette=Layer_color
                  # limit color scale at 99.2% quantile of cell abundance
                  #vmin=0, vmax='p99.2'
                 )


# In[12]:


sc.pl.spatial(adata, cmap='magma',
                  # show first 8 cell types
                  color='windowed log library size',
                  ncols=4, size=1.3,
                  img_key='hires',
                  #palette=Layer_color
                  # limit color scale at 99.2% quantile of cell abundance
                  #vmin=0, vmax='p99.2'
                 )


# plot raw gene expression:

# In[13]:


sc.pl.spatial(adata, cmap='magma',
                  # show first 8 cell types
                  color='IL7R',
                  ncols=4, size=1.3,
                  img_key='hires',
                  #palette=Layer_color
                  # limit color scale at 99.2% quantile of cell abundance
                  #vmin=0, vmax='p99.2'
                 )


# In[14]:


plot_utils.plot_anchor_spots(umap_df,
                             visium_args.pure_spots,
                             visium_args.sig_mean,
                             bbox_x=2
                            )


# ### (3). Optional: Archetypal Analysis
# Overview:
# If users don't provide annotated gene signature sets with cell types, Starfysh identifies candidates for cell types via archetypal analysis (AA). The underlying assumption is that the geometric "extremes" are identified as the purest cell types, whereas all other spots are mixture of the "archetypes". If the users provide the gene signature sets, they can still optionally apply AA to refine marker genes and update anchor spots for known cell types. In addition, AA can identify & assign potential novel cell types / states. Here are the features provided by the optional archetypal analysis:
# - Finding archetypal spots & assign 1-1 mapping to their closest anchor spot neighbors
# - Finding archetypal marker genes & append them to marker genes of annotated cell types
# - Assigning novel cell type / cell states as the most distant archetypes
# 
# Overall, Starfysh provides the archetypal analysis as a complementary toolkit for automatic cell-type annotation & signature gene completion:<br><br>
# 
# 1. *If signature genes aren't provided:* <br><br>Archetypal analysis defines the geometric extrema of the data as major cell types with corresponding marker genes.<br><br>
# 
# 2. *If complete signature genes are known*: <br><br>Users can skip this section and use only the signature priors<br><br>
# 
# 3. *If signature genes are incomplete or need refinement*: <br><br>Archetypal analysis can be applied to
#     a. Refine signatures of certain cell types
#     b. Find novel cell types / states that haven't been provided from the input signature

# #### If signature genes aren't provided
# 
# Note: <br>
# - Intrinsic Dimension (ID) estimator is implemented to estimate the lower-bound for the number of archetypes $k$, followed by elbow method with iterations to identify the optimal $k$. By default, a [conditional number](https://scikit-dimension.readthedocs.io/en/latest/skdim.id.FisherS.html) is set as 30; if you believe there are potentially more / fewer cell types, please increase / decrease `cn` accordingly.

# Major cell types & corresponding markers are represented by the inferred archetypes:<br><br>
# 
# 
# 
# ```Python
# aa_model = AA.ArchetypalAnalysis(adata_orig=adata_normed)
# archetype, arche_dict, major_idx, evs = aa_model.compute_archetypes(r=40)
# 
# # (1). Find archetypal spots & archetypal clusters
# arche_df = aa_model.find_archetypal_spots(major=True)
# 
# # (2). Define "signature genes" as marker genes associated with each archetypal cluster
# gene_sig = aa_model.find_markers(n_markers=30, display=False)
# gene_sig.head()
# ```

# #### If complete signature genes are known
# 
# Users can skip th section & run Starfysh
# 
# #### If signature genes are incomplete or require refinement
# 
# **In this tutorial, we'll show an example of applying best-aligned archetypes to existing `anchors` of given cell type(s) to append signature genes.**

# In[ ]:


aa_model = AA.ArchetypalAnalysis(adata_orig=adata_normed)
archetype, arche_dict, major_idx, evs = aa_model.compute_archetypes(cn=40)
# (1). Find archetypal spots & archetypal clusters
arche_df = aa_model.find_archetypal_spots(major=True)

# (2). Find marker genes associated with each archetypal cluster
markers_df = aa_model.find_markers(n_markers=30, display=False)

# (3). Map archetypes to closest anchors (1-1 per cell type)
map_df, map_dict = aa_model.assign_archetypes(anchors_df)

# (4). Optional: Find the most distant archetypes that are not assigned to any annotated cell types
distant_arches = aa_model.find_distant_archetypes(anchors_df, n=3)


# In[16]:


plot_utils.plot_evs(evs, kmin=aa_model.kmin)


# - Visualize archetypes

# In[17]:


aa_model.plot_archetypes(do_3d=False, major=True, disp_cluster=False)


# - Visualize archetypal - cell type mapping:

# In[18]:


aa_model.plot_mapping(map_df)


# - Application: appending marker genes Append archetypal marker genes with the best-aligned anchors:

# In[ ]:


visium_args = utils.refine_anchors(
    visium_args,
    aa_model,
    #thld=0.7,  # alignment threshold
    n_genes=5,
    #n_iters=1
)

# Get updated adata & signatures
adata, adata_normed = visium_args.get_adata()
gene_sig = visium_args.gene_sig
cell_types = gene_sig.columns


# ## Run starfysh without histology integration
# 
# 
# 
# We perform `n_repeat` random restarts and select the best model with lowest loss for parameter `c` (inferred cell-type proportions):
# 
# ### (1). Model parameters

# In[20]:


import torch
n_repeats = 3
epochs = 200
patience = 50
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')


# ### (2). Model training
# 
# Users can choose to run the one of the following `Starfysh` model without/with histology integration:
# 
# Without histology integration: setting `utils.run_starfysh(poe=False)` (default)
# 
# With histology integration: setting `utils.run_starfysh(poe=True)`

# In[21]:


# Run models
model, loss = utils.run_starfysh(visium_args,
                                 n_repeats=n_repeats,
                                 epochs=epochs,
                                 #patience=patience,
                                 device=device
                                )


# ### Downstream analysis
# 
# ### Parse Starfysh inference output

# In[22]:


adata, adata_normed = visium_args.get_adata()
inference_outputs, generative_outputs,adata_ = sf_model.model_eval(model,
                                                            adata,
                                                            visium_args,
                                                            poe=False,
                                                            device=device)


# ### Visualize starfysh deconvolution results
# 
# **Gene sig mean vs. inferred prop**

# In[31]:


import numpy as np
n_cell_types = gene_sig.shape[1]
idx = np.random.randint(0, n_cell_types)
post_analysis.gene_mean_vs_inferred_prop(inference_outputs,
                                         visium_args,
                                         idx=idx,
                                         figsize=(4,4)
                                        )


# ### Spatial visualizations:
# 
# Inferred density on Spatial map:

# In[24]:


plot_utils.pl_spatial_inf_feature(adata_, feature='ql_m', cmap='Blues')


# **Inferred cell-type proportions (spatial map):**
# 

# In[25]:


def cell2proportion(adata):
    adata_plot=sc.AnnData(adata.X)
    adata_plot.obs=utils.extract_feature(adata_, 'qc_m').obs.copy()
    adata_plot.var=adata.var.copy()
    adata_plot.obsm=adata.obsm.copy()
    adata_plot.obsp=adata.obsp.copy()
    adata_plot.uns=adata.uns.copy()
    return adata_plot
adata_plot=cell2proportion(adata_)


# In[26]:


adata_plot


# In[27]:


sc.pl.spatial(adata_plot, cmap='Spectral_r',
                  # show first 8 cell types
                  color=['Basal','LumA','LumB'],
                  ncols=4, size=1.3,
                  img_key='hires',
                  vmin=0, vmax='p90'
                 )


# In[28]:


ov.pl.embedding(adata_plot,
               basis='z_umap',
                color=['Basal', 'LumA', 'MBC', 'Normal epithelial'],
               frameon='small',
                vmin=0, vmax='p90',
                cmap='Spectral_r',
               )


# In[29]:


pred_exprs = sf_model.model_ct_exp(model,
                                   adata,
                                   visium_args,
                                   device=device)


# Plot spot-level expression (e.g. `IL7R` from *Effector Memory T cells (Tem)*):
# 

# In[30]:


gene='IL7R'
gene_celltype='Tem'
adata_.layers[f'infer_{gene_celltype}']=pred_exprs[gene_celltype]

sc.pl.spatial(adata_, cmap='Spectral_r',
                  # show first 8 cell types
                  color=gene,
                  title=f'{gene} (Predicted expression)\n{gene_celltype}',
                  layer=f'infer_{gene_celltype}',
                  ncols=4, size=1.3,
                  img_key='hires',
                  #vmin=0, vmax='p90'
                 )


# ### Save model & inferred parameters

# In[ ]:


# Specify output directory
outdir = './results/'
if not os.path.exists(outdir):
    os.mkdir(outdir)

# save the model
torch.save(model.state_dict(), os.path.join(outdir, 'starfysh_model.pt'))

# save `adata` object with inferred parameters
adata.write(os.path.join(outdir, 'st.h5ad'))