A newer version of the Gradio SDK is available:
5.21.0
title: In-demand ML skills
emoji: 💻
colorFrom: yellow
colorTo: purple
sdk: gradio
sdk_version: 5.0.1
app_file: app.py
pinned: false
short_description: Monitoring in-demand skills on ML job-postings
In-demand Skill Monitoring for Machine Learning Industry
About
This projects aims to monitor in-demand skills for machine learning roles. Skills are extracted with a BERT-based skill extraction model called JobBERT, which is continously fine-tuned on the job postings. The skills are monitored/visualized by: 1. embedding the extracted skills tokens into vector form, 2. performing dimensionality reduction with UMAP, 3. visualizing the reduced embeddings.
Monitoring Platform Link
Architecture & Frameworks
- Hugging Face Spaces: Used as an UI to host interactive visualisation of skills embeddings and their clusters.
- GitHub Actions: Used to schedule training, inference and visualisation-updating scripts.
- Rapid API: The API used to scrape job descriptions from LinkedIn
- Weights & Biases: Used for model training monitoring as well as model storing.
- OpenAI API: Used to extract ground-truth from job descriptions by leveraging multi-shot learning and prompt engineering.
High-Level Overview
Models
- BERT - finetuned skill extraction model, lightweight.
- LLM - gpt-4o for skill extraction with multi-shot learning. Computationally expensive.
- Embedding model - SentenceTransformers used to embed skills into vectors.
- spaCy - sentence tokenization model.
Pipeline
Pipeline
The follow scripts are scheduled to automate the skill monitoring and model tranining processes continually.

1. Job-posting scraping
Fetching job descriptions for machine learning from LinkedIn via Rapid API.
2. Skills tagging with LLM
We opinionately extract the ground truth of skills from the job descriptions by leveraging multi-shot learning and prompt engineering.
3. Model training
The skill extraction model is finetuned with respect to the extracted groundtruth.
4. Skills tagging with JobBERT
Skills are extracted from job-postings with finetuned model
5. Embedding & visualization
Extracted skills are embedded, reduced and clustered with an embedding model, UMAP and K-means respectively.
Job Scraping
This component scrapes job descriptions from the LinkedIn Job Search API for Machine Learning, and saves them in text files for further analysis.
Workflow
API Configuration:
- The script uses the
linkedin-job-search-api.p.rapidapi.com
endpoint to fetch job data. - API access is authenticated using a RapidAPI key stored as an environment variable
RAPID_API_KEY
.
- The script uses the
Data Retrieval:
- The script fetches jobs matching the keyword
machine learning
. - It retrieves job details including the description, which is saved for further analysis.
- The script fetches jobs matching the keyword
Job Description Extraction:
- Each job description is saved in a
.txt
file under thejob-postings/<date>
folder.
- Each job description is saved in a
Skill Embeddings and Visualization
We generate embeddings for technical skills listed in .txt files and visualizes their relationships using dimensionality reduction and clustering techniques. The visualizations are created for both 2D and 3D embeddings, and clustering is performed using KMeans to identify groups of similar skills.
Workflow
1. Input Data
- Skills are loaded from
.txt
files located in date-based subfolders under the./tags
directory. - Each subfolder corresponds to a specific date (e.g.,
03-01-2024
).
2. Embedding Generation
- The script uses the
SentenceTransformer
model (paraphrase-MiniLM-L3-v2
) to generate high-dimensional embeddings for the unique skills.
3. Dimensionality Reduction
- UMAP (Uniform Manifold Approximation and Projection) is used to reduce the embeddings to:
- 2D: For creating simple scatter plots.
- 3D: For interactive visualizations.
4. Clustering
- KMeans clustering is applied to the 3D embeddings to group similar skills into clusters.
- The number of clusters can be specified in the script.
5. Visualization and Outputs
- 2D Projection: Saved as PNG images in the
./plots
folder. - 3D Projection: Saved as interactive HTML files in the
./plots
folder. - 3D Clustering Visualization: Saved as HTML files, showing clusters with different colors.