Tilde Logo

Tilde Fellowship Program

TL;DR

  • •We're launching the Tilde Fellowship to support research in reverse-engineering important phenomena in pretraining science (arch, optimizers, etc).
  • •We will provide GPU compute, direct mentorship, and collaboration with Tilde staff, along with opportunities to match with other fellows.
  • •You do not need prior experience in mechanistic research to apply. Applications are reviewed on a rolling basis, with an intended start date in mid-October.
  • •Apply here:
    Apply Now

Overview

We're launching the Tilde Fellowship to support research in the foundational science of deep learning.

Much of modern ML progress has come from scaling models and empirically optimizing architectures. This has been astonishingly effective, but it often leaves us with only a surface-level grasp of why models work, or how to systematically extend their capabilities.

At Tilde, we believe mechanistic understanding is the foundation for entirely new architectures, capabilities, and safety tools. Endlessly throwing tricks and compute at problems can get you to the next benchmark, but it rarely tells you why something works or how to build on it. The most powerful breakthroughs will come from deeply understanding how models work.

Through the fellowship, we want to support, collaborate with, and learn from researchers pushing this frontier. We are especially excited by ambitious projects that may not have a high chance of "success." A principled technical report detailing techniques or methods that "did not work" is still a valuable contribution.

Program Details

Who Should Apply

We'll work side-by-side with fellows, providing compute, mentorship, and direct collaboration with our technical team, along with community support. Applicants may (but don't have to) fall into three broad categories:

  1. 1.Have Idea + Team, No Resources: If you have an idea and a team, but want resources and/or mentorship, then we can provide the compute and guidance to carry out the project.
  2. 2.Have Idea, No Team/Resources: If you already have an idea but no team, we'll pair you with like-minded applicants and work with you directly to shape and execute the project.
  3. 3.No Idea, No Team/Resources: We will collaborate with and connect you with projects that will serve both as a strong learning experience and an opportunity to contribute to novel, impactful research.

You do not need prior experience in mechanistic research to apply. Applications are reviewed on a rolling basis, with an intended start date in mid-October.

What We're Looking For

Projects should be in foundational science of deep learning and include a mechanistic understanding component. This might involve using techniques from:

  • •Activation/Weight Geometry
  • •Causal Analysis
  • •Patching and Interventions
  • •Learning Dynamics

To understand phenomena in:

  • •Architecture, Test-Time Learning, & Long-Context Sequence Modeling
  • •Optimizers and Training Algorithms
  • •Quantization, Pruning, Sparsity, and Weight Sharing

We are not looking for small incremental tweaks, benchmark maxing, or high-level model orchestration. We are more interested in supporting projects that lead to deeper understanding of how models (should) work. The criterion for success is not performance improvement, rather:

  • •Emphasis on rigorous science
  • •Advance our fundamental understanding of why an approach does or does not work
  • •In particular, we believe negative results can be even more compelling stories than positive ones!

Example Past Publications

We provide here some previous publications as few-shot learning examples (we claim no affiliation with most of the papers listed below).

Example Directions

  • •An analysis of the unreasonable effectiveness of forget gates across architectures.
  • •On the sensitivity of different optimizers to data ordering.
  • •Theoretical vs empirical storage capacity of gated linear units and their nonlinearities.
  • •The effect of switching optimizers during pretraining on circuit formation

Logistics & Expectations

  • •Time commitment: Minimum of 20–30 hrs/week.
  • •Support: We provide GPU compute, direct mentorship, and collaboration with Tilde staff, along with opportunities to match with other fellows. (No direct stipends; we want to maximize compute and collaboration.)
  • •Community: Fellows will collaborate directly with Tilde technical staff through weekly meetings, joint experiments, and feedback sessions, and will present results together at program end.
  • •Extras: We will fly fellows out at completion to present research on-site (and there will be Tilde merch)!

Apply

Fill out this application with:

  • •Proposed duration and scope
  • •Compute needs (ballpark GPU ask is fine)
  • •Any teammates (or interest in matching)
  • •Background and motivations

Applications are reviewed on a rolling basis, and we'll get back to you quickly.

Apply Now
Tilde Logo
AboutJoinBlogChallenges