What one matrix changes.
Google’s Magenta RealTime 2 already knows how to generate piano and flute. We change one small part that carries style information into its music generator.
The equations explain how to make that change depend on the input, then store the entire operation in the weights.
Start at the top and try the controls as you go. The interactive numbers are deliberately tiny teaching examples, not measured Magenta activations. The real dimensions, calibration data and patch facts are stated separately.
1Inputs change the request.
Weights change the rule.
A neural network processes numbers. Its weights are the numbers that determine how it processes them. A checkpoint is a saved collection of those weights.
| Thing | What it represents | When it changes |
|---|---|---|
| Input values | The current request | With each prompt or reference recording |
| Weights | The rules for processing the request | During training, or when explicitly edited |
Suppose a tiny model calculates output = 2a + 3b. The values a and b are inputs; 2 and 3 are weights. A new input gives a new answer using the same rule. Editing a weight changes the rule itself.
Try both kinds of change
Tiny arithmetic exampleKeep a and b fixed, then move a weight. You are changing the answer to the same request.
Our project changes a rule inside Magenta and saves that changed rule.
What does “without backpropagation” mean?
During ordinary neural-network training, a model makes predictions, a loss measures their errors, and backpropagation calculates how the loss changes with each weight. An optimizer then adjusts weights, and the process repeats.
This project calculates a replacement matrix directly from encoded examples and linear equations. It still uses data and solves an optimization problem, but it does not use gradients to iteratively train Magenta’s weights.
2How words become a musical instruction
The prompt “A solo honky-tonk piano plays a fast boogie-woogie, only piano” reaches the generator through a numerical representation. This is the style path relevant to our edit:
A vector is an ordered list of numbers. Here it describes aspects of the request, including instrument, texture and arrangement. The entries do not come with labels such as “piano” or “flute”; information is distributed across the list.
A matrix is a rectangular table of numbers. The adapter uses its weights to combine the 768 input values into 1,024 output values. For output number 1, it calculates:
It repeats this for all 1,024 outputs. The compact notation is xᵀW. The superscript ᵀ means transpose: write the column vector horizontally so the matrix multiplication fits. This expression gives a row of output values.
We chose this adapter because it carries style information into the generator. Its architectural role identifies a useful place to edit; calibration examples determine the numerical change.
An audio reference guides the style of new music. This is not an effect that replaces an instrument in an existing recording note for note.
3Change what “piano” leads to
We want a piano-related input to send a more flute-like signal into the generator, while other inputs receive little of that change.
| Input | Desired effect |
|---|---|
| Piano | Shift the conditioning toward flute. |
| Guitar or drums | Keep approximately the original conditioning. |
| Flute | Keep approximately the original conditioning. |
There are two separate questions. What change should we add? An output direction, d. How much should this input receive? A gate response, s(x).
The direction is fixed after the patch is built. The response varies because each input is different. Both ingredients are eventually combined into the saved matrix.
4A direction is a difference
Start with ordinary coordinates. To move from P = [2, 1] to F = [1, 3], add their difference:
The same idea works with many more coordinates. In the model, we encode piano descriptions and average their vectors, then do the same for flute. Averaging reduces dependence on one description or one style-mapper seed.
A vector average works coordinate by coordinate: the average of [1, 2] and [3, 4] is [2, 3]. The symbol μ denotes such a group average.
Move along the difference
Two illustrative coordinatesHalfway from the piano point to the flute point.
This picture explains vector addition. Distance on this drawing does not measure real musical quality or instrument identity.
The averages, μpiano and μflute, each have 768 entries. We need an output shift with 1,024 entries, so we translate their difference through the original adapter:
Because the adapter is linear, this is also the average flute conditioning minus the average piano conditioning. Subtract before translating, or translate both and subtract: the answer is the same.
Find the generator’s average conditioning signal for flute and subtract its average signal for piano. The difference is the change we want to add.
In the project, this direction comes from an earlier paired text calibration. It is an empirical direction, not a guaranteed “instrument-only” axis. Texture, arrangement and timbre can overlap.
5How much change should this input get?
A constant flute shift would affect every request. Instead, we calculate a vector g with 768 entries. Each coefficient controls how much its corresponding input value contributes to one final score.
The same fixed g gives different scores because each request has a different x. Contributions can reinforce or cancel each other.
We call this a gate response, but it is not a separate neural network, a probability or an on/off switch. It is a linear score. We aim for one on piano examples and zero on controls.
For the interactive matrix later, we use three invented input coordinates. They have no real musical labels:
| Toy input | Vector x | Desired score |
|---|---|---|
| Piano | [1, 1, 0] | 1 |
| Guitar | [1, −1, 0] | 0 |
| Flute | [0, 0, 1] | 0 |
Notice that piano and guitar share a positive first coordinate. A single coordinate does not identify the instrument; the combination matters.
6Turn the desired behavior into equations
For our three-coordinate toy, call the unknown gate coefficients a, b and c. The requested responses give three equations:
The second equation says a = b. The first then says 2a = 1. So g = [0.5, 0.5, 0].
Check it: piano gives 0.5 + 0.5 = 1. Guitar gives 0.5 − 0.5 = 0. Flute gives 0. The coefficients were calculated directly from the behavior we requested.
The real gate uses the same idea, with 768 unknown coefficients and calibration examples from the frozen encoder.
| Real calibration data | How it is used |
|---|---|
| 432 text encodings | Nine instruments × 12 descriptions × four style-mapper seeds |
| Nine synthetic audio references | Three piano examples and six controls |
These produce 16 required responses: the piano text mean and piano audio mean must score one; eight other instrument means and six individual audio controls must score zero.
The requirements apply to those particular calibration vectors. They do not promise that every individual piano example, or every new recording, will produce exactly one.
7Choose a gate that behaves consistently
The toy had three unknowns and three equations. The real problem has 768 unknowns and only 16 required responses. Many possible gates can satisfy the constraints.
Some would behave badly. A piano mean could score one even if individual piano descriptions receive wildly different scores whose average happens to be one.
We therefore penalize variation across the piano examples, nonzero responses to text controls, and very large coefficients. The audio controls are protected by the explicit constraints. A matrix called Q packages these quadratic penalties.
Read the full gate formula, one part at a time
| Symbol | Meaning |
|---|---|
| A · 16 × 768 | One calibration constraint per row |
| b · 16 values | The two desired ones and fourteen zeros |
| Q · 768 × 768 | The cost of unstable responses and large coefficients |
| g · 768 values | The gate we are solving for |
| ⁻¹ | A matrix inverse in the mathematical expression; the code uses a linear solve |
| ⁺ | A pseudoinverse, a generalized inverse that can handle redundant constraints |
This is the direct solution to the constrained minimization above. It is not a second model or another training procedure.
The implementation first solves QB = Aᵀ. It then computes g = B(AB)⁺b. This is the same formula without explicitly constructing Q’s inverse. It checks that Ag is within numerical tolerance of b, and rejects inconsistent or ill-conditioned calibration.
Q includes centered piano-text variation, centered piano-audio variation, squared responses to text controls, and a scaled identity term that regularizes the coefficients.
You do not need to memorize the inverse expression to explain the method. The important point is the problem it solves: meet the stated responses while discouraging unstable behavior.
This is fitting from calibration data. Python calculates the coefficients through linear algebra; it does not repeatedly backpropagate through Magenta or take optimizer steps. The encoder and generator remain frozen while the examples are collected.
8Store “when” and “what” in one matrix
We now have the input coefficients g and output direction d. For every input position i and output position j, set the weight change to:
The outer product gdᵀ makes the whole table at once. A 768 × 1 column multiplied by a 1 × 1,024 row produces a 768 × 1,024 matrix, exactly the shape we need.
We are not choosing 786,432 independent fixes. The update is built from one 768-number gate and one 1,024-number direction. Every input’s additional output lies along the same direction d, with a different multiplier. This is why the update is rank one, before FP32 rounding.
A complete edit you can inspect
3 inputs × 2 outputsInvented teaching model
This toy uses the gate we solved above: g = [0.5, 0.5, 0]. Its piano conditioning is [2, 1], flute conditioning is [1, 3], and therefore d = [−1, 2].
Original W fixed
Change ΔW αgdᵀ
Edited W′ W + ΔW
Hover or select a changed weight to inspect its calculation. Rose: decrease. Blue: increase.
Two calculations, the same answer
Changing x changes the response, not the matrices. Changing α changes the saved matrix. The real patch fixes α at 1.4; this control is only for learning.
9The ordinary multiplication does it all
Look at just one output value, j. The extra contribution from the edited weights is the sum of each input multiplied by its corresponding weight change:
For this output, α and dj are fixed. Pull them outside the sum. What remains is exactly the dot product that defines the gate:
For all outputs together, that is the second README equation:
The left side is what the model actually runs: multiply the input by the saved adapter. The right side explains the effect. They are two ways of writing the same calculation.
There is no separate “detect piano, then add flute” program during generation. Multiplication by the edited weights already performs that operation.
10Strength is chosen. Selectivity is imperfect.
The main patch uses α = 1.4, chosen through development comparisons and listening. It is an experimental choice, not an optimum proved by the equations.
| Gate response | Added shift at α = 1.4 | Effect |
|---|---|---|
| 1 | 1.4d | Full intended strength |
| 0 | 0 | No added shift |
| 0.5 | 0.7d | A partial shift |
| −0.5 | −0.7d | The direction reverses |
These scores illustrate the calculation; they are not measurements of the demo clips. A linear gate can produce values below zero or above one. There is no automatic clamp.
A recording containing piano and strings arrives as a combined style representation. The gate does not isolate the piano track. An unfamiliar combination can receive too little change, the wrong change, or a negative response.
In the actual evaluations, some mixed arrangements failed. A piano-and-strings reference produced a negative gate response, reversing the intended change. The newer experimental patch adds four zero-response constraints, but its gate is still linear and unbounded.
Two blind quality comparisons against simply prompting the original model for flute split 1–1, with one listener. The project demonstrates selected audible changes saved in the weights; it does not establish a quality advantage over prompting or reliable preservation of every other instrument.
11Save the new rule, then use the model normally
Keep the editing stage separate from generation. Direction and gate are calculated while building the patch. During generation, their combined effect is already in W′.
Building the patch
- Run frozen encoders on calibration examples.
- Calculate the output direction d.
- Solve for the gate g.
- Choose the strength through comparisons.
- Calculate W′ and save it.
Every new request
- Encode the prompt or reference.
- Multiply its style vector by W′.
- Generate and decode the music.
No recalibration, hidden flute prompt or activation hook.
The patch stores the complete replacement matrix. Its size follows directly from its shape and number format:
The full checkpoint contains 2,459,163,592 parameters, including the codec. The replaced matrix is about 0.032% of that total. The other 617 tensors are unchanged.
That percentage counts affected values; it is not the size of each numerical change. Within the adapter, the update’s Frobenius norm is 5.026% of the original matrix’s norm. The small global fraction can matter because this matrix carries style information into the generator.
| Verification | What it establishes |
|---|---|
| Compare checkpoint tensors | Only the intended matrix changed. |
| Rebuild from saved calibration tokens | The recalculated patch has the same SHA-256 hash. |
| Reload in a fresh process | One development recording reproduced byte for byte on the tested runtime. |
| Matched prompts, seeds and settings | Before-and-after generations isolate the checkpoint change under those settings. |
These checks establish properties of the saved artifact and tested runs. Listening comparisons are still needed to judge instrument identity, sound quality and accompaniment.
Explain it to someone else
“Google’s Magenta already knows how to generate piano and flute. We used examples to calculate a shift between their internal style representations, and a score that makes this shift respond mainly to piano inputs. We combined those calculations into one replacement matrix and saved it into the model. The same piano input can then produce flute through the original generation code, without backpropagation or an extra runtime instruction. Keeping the change selective on unfamiliar or mixed-instrument inputs is the difficult part.”
Where the WSP part comes in
This is a human-guided, tool-assisted weight-space programming experiment. GPT-6 Astra designed and implemented the main edit using Python and frozen encoder examples. I set the goal and supplied listening feedback.
The mechanism is a calibration-based rank-one update using established linear algebra. Magenta’s flute-generating ability already existed; the edit changes when that ability is invoked. It does not imply that someone identified individual “piano weights” by looking at raw numbers.
Back to the beginning