Back to stories

Comments

10 total · 10 loaded

Ranked by HN
quirino8hon HN
I've been studying these a bunch for a project in university. Last week I went over the derivation of the ELBO for a couple hours and it was a very fun and elucidating exercise.

Once you give names to the larger mathematical structures and understand them a bit better it becomes quite simple. I wish some of the blogs/papers I'd read had named "Importance Sampling".

The probability notation can be pretty confusing too. Sometimes it's hard to understand the "types" of some variables. But I'm inexperienced.

ChatGPT was surprisingly helpful. If you put in the work to truly understand the where the gaps are in your mental model (which parts aren't completely intuitive), it can do an amazing job filling in the gaps.

radarsat14hon HN
Something I've wondered, maybe I should just do it if I can find some time, but... given DeepSeek's nice results on using rendered text as input, I'm wondering if anyone has given serious research efforts towards image-based diffusion methods for text.

As in, instead of all the complexities induced by discrete token generation, just generate the image of the text using standard image diffusion methods, then convert it to text.

If you used a single, monospace font, I bet this would be even pretty efficient, because the OCR problem becomes basically just direct template matching.

But I guess probably there is already a paper out there, I haven't searched. I'd be curious to know if it compares on par with token-based methods.

amelius3hon HN
I've been wondering why we can't skip the entire token embedding step and just feed the model raw Unicode. I suppose the reason for that is efficiency and it's related to your question. Makes one wonder what other steps we can do more efficiently in code than in neurons.
p1esk1hon HN
There’s has a ton of work on character or byte level encodings for llms. The problem is you expand your input tokens by 3-4x. Expensive.

Also, you still need token embeddings (I think you might be confused how that works).

amelius1hon HN
Could be! I have not (yet) spent much time learning about how llms work, just the occasional blog here and there. My main question is why we need a bit of additional code to massage the input into tokens and why the neural network cannot do it, i.e., let the embedding be a latent space that forms naturally when training the network. If that makes sense.
archerx3hon HN
I think it would make training take much much longer but the models would be able to tell you how many Rs are in strawberry.
amelius3hon HN
Yes. By the way I think it is a pity that all the tutorials start with token embedding; by teaching LLMs like that, you don't learn why they are necessary.
gdiamos6hon HN
I’d like to see more of these models.

I’ve been using diffusion Gemma and it is very fast on GPUs in output token/sec.

In the diffusion Gemma whitepaper, they say they could have done better with more time and compute.

Even with those caveats, it is very uses-able as a local model.

electroglyph9hon HN
good stuff, no mention of confidence tho, recommend having a look at diffusiongemma and others.