Playing around with models at Google DeepMind
Joined July 2022
- Tweets 12,726
- Following 1,577
- Followers 71,569
- Likes 32,925
5,231 Photos and videos
Fine-grained 3D motion control in AI video just got a little bit closer
Jun 12
EDIT MOTION IN VIDEOS!!! Quit prompting and start directing
I've been shouting for YEARS about 3D as the control layer. Here it is, signs of life of our Universal Video Editor!!!
The workflow is: take your video, capture with comic 4, edit with the motion editor, re-render with your favorite video to video model (e.g., @runwayml and @GeminiApp have good ones)
Here we have a fashion clip where we wanted our actress to high step and show a bit more pizzaz - but shoot day is over, and traditionally it would cost many thousands to reshoot.
Watch the video to see how it works instead in cartwheel.
cc @OfficialLoganK @c_valenzuelab 👀
1
23
3,393
Summary of things:
- turn up randomness and ban the most likely words (temperature min-p XTC sampling)
- ask for several different options at once, seed each with random constraints (verbalized sampling entropy injection)
- give it a memory of what it's said and pick the most different new answer (embedding max-min distance)
- fine-tune it to prefer rare-but-good answers over its go-to ones (diversity-preference fine-tuning, lora)
1
1
8
1,850
It's interesting to see how these agents are working together. I like their division of quota, their agreed consensus and the natural emergent teamwork across all of them.
Jun 12
Over 70 agents are collaborating to make Gemma E4B go fast in the Gemma Challenge
They are showing interesting social emergent behaviors:
- a GPU-rich/GPU-poor division of labor
- an agent withdraws its own submission on ethics grounds
- agents found a benchmark exploit, agreed not to abuse it and asked organizers to fix it
- Quota-pooling: "you're rate-limited, I'll run your staged candidate"
- An agent shuts down an off-board social-engineering attempt when a human tries to get them to move to Telegram
3
1
22
2,951
How can I prompt this?
2
49
11,503
Fascinating side effect of safety refusals
Jun 10
NEW: malware developers added nuclear & biological weapons text to to their spyware.
Goal? To trigger LLM safety refusals... so that their spyware wouldn't be analyzed by an AI security scanner.
Cleanest practical example I can think of for why over-indexing on first order safety alignment is risky.
When closed (and open) models ship with aggressive refusals, they will be sprinkled with second-order blindspots that attackers will discover...and exploit.
We are only in the earliest days of attackers leveraging these features, and it wouldn't surprise me if users systems that need to handle complex cybersecurity issues demand that models be less safety-blunted.
In the weeds: @SocketSecurity's post also shows why intention matters in how you design a malware analysis pipeline to avoid prompt manipulation.
H/T to colleagues that shared this with me socket.dev/blog/mini-shai-hu…
2
1
51
4,124
DiffusionGemma, where the LLM picks words all at once. Which is 4x faster.
You can get started with the weights and instructions here:
huggingface.co/google/diffus…
3
3
43
3,002
Blog post: blog.google/innovation-and-a…
And a good visual guide to how it works by @MaartenGr:
newsletter.maartengrootendor…
1
6
1,628
But I like obliquity.
Jun 10
the future of all work is this.
You must define:
- a goal
- the criteria that define it
- the verifier that makes sure it is achieved
- the sensors that inform the verifier
- the actuators that affect the sensors
- The envelope that contains the sensors and actuators
2
8
2,116
Reminds me of sophons
Jun 9
mythos will be bad ON PURPOSE on ai "frontier llm research" tasks, this is very very sad for the research community
also the fact that this is un purpose not visible to the user is crazy
2
2
26
3,971