Smash the like button if you’re also excited to see break down on how to build the Deepseek R1 from scratch!🚀     See Less

I am following your work in C++ now in addition to Raff K. I think you would love to render float space, I do this.     See Less

Thank you, thanks a lot Andrej, for providing such a course for free, learnt a lot from this course, and am very thankful to you :)     See Less

I'm training my GPT2 model on an old 730 card with 2GB ram
It will also run a pre-trained GPT2 LLM out of the box, but you have to use/install python3.7 and CUDA 9.2 to support it.
    See More you've ever taken a backprop class in college, you can use "gradient accumulation" to effectively have a minibatch size of any size you want, with the small memory footprint of a single sample.    See Less

Thanks!     See Less

Hello Andrej, thank you so much for creating and sharing such high quality resources. It's really impressive and super helpful.     See Less

Andrej! Multimodal. RAG! 🎉 🙏🙏🤗     See Less

If anyone is following from nn-zero-to-hero implementation and confused about the reason to transpose these openai's weight matrices  [attn.c_attn.weight', 'attn.c_proj.weight&     See More ;mlp.c_fc.weight], here's an explanation.

In previous videos, we simply defined our own Linear class as

class Layer:
def __init__(self,fan_in, fan_out, bias=False):
self.w = torch.randn((fan_in, fan_out),generator = g) #/ (fan_in)**(0.5) # applying kaiming init
self.bias = bias
if bias:
self.b = torch.zeros(fan_out)

def __call__(self, x):
y = x @ self.w
self.out = y + self.b if self.bias else y
return self.out


But this is a slight deviation from pytorch's implementation, If you go to pytorch's nn.Linear documentation, the weight matrix is constructed with shape (out_features,in_features) and then while applying this weight matrix it is simply transposed y=x@W.T + b.

which is exactly the reverse of what we did in our previous implementation.

so the proper way is to adhere to pytorch's implementation i.e if we define linear layer nn.Linear(fan_in,fan_out) our weight will have shape (fan_out,fan_in)    See Less

The fact that the number of current subs and views are pretty close is driving me crazy!     See Less

God bless you     See Less