Guardians of the Agentic System: Preventing Many Shots Jailbreak with Agentic System
Paper
•
2502.16750
•
Published
•
10
Ksenia Se
Kseniase
AI & ML interests
None yet
Recent Activity
reacted
to
their
post
with 🚀
1 day ago
8 types of RoPE
As we always use Transformers, it's helpful to understand RoPE—Rotary Position Embedding. Since token order matters, RoPE encodes it by rotating token embeddings based on their position, so the model knows how to interpret which token comes first, second, and so on.
Here are 8 types of RoPE that can be implemented in different cases:
1. Original RoPE -> https://huggingface.co/papers/2104.09864
Encodes token positions by rotating token embeddings in the complex plane via a position-based rotation matrix, thereby providing the self-attention mechanism with relative positional info.
2. LongRoPE -> https://huggingface.co/papers/2402.13753
Extends the context window of pre-trained LLMs to 2048k tokens, leveraging non-uniformities in positional interpolation with an efficient search.
3. LongRoPE2 -> https://huggingface.co/papers/2502.20082
Extends the effective context window of pre-trained LLMs to the target! length, rescaling RoPE guided by “needle-driven” perplexity.
4. Multimodal RoPE (MRoPE) -> https://huggingface.co/papers/2502.13923
Decomposes positional embedding into 3 components: temporal, height and width, so that positional features are aligned across modalities: text, images and videos.
5. Directional RoPE (DRoPE) -> https://huggingface.co/papers/2503.15029
Adds an identity scalar, improving how angles are handled without extra complexity. It helps balance accuracy, speed, and memory usage.
6. VideoRoPE -> https://huggingface.co/papers/2502.05173
Adapts RoPE for video, featuring 3D structure, low-frequency temporal allocation, diagonal layout, and adjustable spacing.
7. VRoPE -> https://huggingface.co/papers/2502.11664
An another RoPE for video, which restructures positional indices and balances encoding for uniform spatial focus.
8. XPos (Extrapolatable Position Embedding) -> https://huggingface.co/papers/2212.10
Introduces an exponential decay factor into the rotation matrix, improving stability on long sequences.
Organizations
Kseniase's activity
upvoted
5
papers
23 days ago
Towards an AI co-scientist
Paper
•
2502.18864
•
Published
•
44
SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution
Paper
•
2502.18449
•
Published
•
71
Beyond Release: Access Considerations for Generative AI Systems
Paper
•
2502.16701
•
Published
•
12
Slamming: Training a Speech Language Model on One GPU in a Day
Paper
•
2502.15814
•
Published
•
68
upvoted
4
papers
25 days ago
Gödel Agent: A Self-Referential Agent Framework for Recursive Self-Improvement
Paper
•
2410.04444
•
Published
•
2
Tree of Thoughts: Deliberate Problem Solving with Large Language Models
Paper
•
2305.10601
•
Published
•
12
Chain of Hindsight Aligns Language Models with Feedback
Paper
•
2302.02676
•
Published
•
1
ReAct: Synergizing Reasoning and Acting in Language Models
Paper
•
2210.03629
•
Published
•
24