Magenta, explained.

What one matrix changes.

Google’s Magenta RealTime 2 already knows how to generate piano and flute. We change one small part that carries style information into its music generator.

The equations explain how to make that change depend on the input, then store the entire operation in the weights.

Start at the top and try the controls as you go. The interactive numbers are deliberately tiny teaching examples, not measured Magenta activations. The real dimensions, calibration data and patch facts are stated separately.

1Inputs change the request.
Weights change the rule.

A neural network processes numbers. Its weights are the numbers that determine how it processes them. A checkpoint is a saved collection of those weights.

ThingWhat it representsWhen it changes
Input valuesThe current requestWith each prompt or reference recording
WeightsThe rules for processing the requestDuring training, or when explicitly edited

Suppose a tiny model calculates output = 2a + 3b. The values a and b are inputs; 2 and 3 are weights. A new input gives a new answer using the same rule. Editing a weight changes the rule itself.

Try both kinds of change

Tiny arithmetic example
2 × 1 + 3 × 2 = 8

Keep a and b fixed, then move a weight. You are changing the answer to the same request.

Our project changes a rule inside Magenta and saves that changed rule.

What does “without backpropagation” mean?

During ordinary neural-network training, a model makes predictions, a loss measures their errors, and backpropagation calculates how the loss changes with each weight. An optimizer then adjusts weights, and the process repeats.

This project calculates a replacement matrix directly from encoded examples and linear equations. It still uses data and solves an optimization problem, but it does not use gradients to iteratively train Magenta’s weights.

2How words become a musical instruction

The prompt “A solo honky-tonk piano plays a fast boogie-woogie, only piano” reaches the generator through a numerical representation. This is the style path relevant to our edit:

Text prompt or reference recordingThe input stays the same in the before-and-after comparison.
Frozen MusicCoCa style encoderStyle tokens select codebook vectors; their sum produces x.
768-number style vector, xAn ordered list carrying information about the requested style.
Style adapter, WA 768 × 1,024 matrix. This is the only tensor replaced.
1,024-number conditioning signalA numerical input that guides the music generator.
Music generator and audio decoderNewly generated audio.
The relevant style path, not the complete model architecture.

A vector is an ordered list of numbers. Here it describes aspects of the request, including instrument, texture and arrangement. The entries do not come with labels such as “piano” or “flute”; information is distributed across the list.

A matrix is a rectangular table of numbers. The adapter uses its weights to combine the 768 input values into 1,024 output values. For output number 1, it calculates:

y1=x1W1,1+x2W2,1+⋯+x768W768,1
Multiply each input by its weight, then add the products.

It repeats this for all 1,024 outputs. The compact notation is xᵀW. The superscript ᵀ means transpose: write the column vector horizontally so the matrix multiplication fits. This expression gives a row of output values.

We chose this adapter because it carries style information into the generator. Its architectural role identifies a useful place to edit; calibration examples determine the numerical change.

An audio reference guides the style of new music. This is not an effect that replaces an instrument in an existing recording note for note.

3Change what “piano” leads to

We want a piano-related input to send a more flute-like signal into the generator, while other inputs receive little of that change.

InputDesired effect
PianoShift the conditioning toward flute.
Guitar or drumsKeep approximately the original conditioning.
FluteKeep approximately the original conditioning.

There are two separate questions. What change should we add? An output direction, d. How much should this input receive? A gate response, s(x).

The direction is fixed after the patch is built. The response varies because each input is different. Both ingredients are eventually combined into the saved matrix.

4A direction is a difference

Start with ordinary coordinates. To move from P = [2, 1] to F = [1, 3], add their difference:

F − P = [−1, 2]
[2, 1] + [−1, 2] = [1, 3]

The same idea works with many more coordinates. In the model, we encode piano descriptions and average their vectors, then do the same for flute. Averaging reduces dependence on one description or one style-mapper seed.

A vector average works coordinate by coordinate: the average of [1, 2] and [3, 4] is [2, 3]. The symbol μ denotes such a group average.

Move along the difference

Two illustrative coordinates
A shift from piano conditioning P to flute conditioning F P is at 2, 1. F is at 1, 3. The moving point follows P plus the selected amount times minus 1, 2. These are invented coordinates. Conditioning coordinate 1Coordinate 2 P [2, 1] F [1, 3]
Piano averageFlute averageShifted point
P + 0.5d = [1.5, 2]

Halfway from the piano point to the flute point.

This picture explains vector addition. Distance on this drawing does not measure real musical quality or instrument identity.

The averages, μpiano and μflute, each have 768 entries. We need an output shift with 1,024 entries, so we translate their difference through the original adapter:

d=W⊤(μflute−μpiano)
Here d is a column: Wᵀ has 1,024 rows and 768 columns.

Because the adapter is linear, this is also the average flute conditioning minus the average piano conditioning. Subtract before translating, or translate both and subtract: the answer is the same.

Find the generator’s average conditioning signal for flute and subtract its average signal for piano. The difference is the change we want to add.

In the project, this direction comes from an earlier paired text calibration. It is an empirical direction, not a guaranteed “instrument-only” axis. Texture, arrangement and timbre can overlap.

5How much change should this input get?

A constant flute shift would affect every request. Instead, we calculate a vector g with 768 entries. Each coefficient controls how much its corresponding input value contributes to one final score.

s(x)=x⊤g=∑i=1768xigi
Multiply corresponding entries and add the products: a dot product.

The same fixed g gives different scores because each request has a different x. Contributions can reinforce or cancel each other.

We call this a gate response, but it is not a separate neural network, a probability or an on/off switch. It is a linear score. We aim for one on piano examples and zero on controls.

For the interactive matrix later, we use three invented input coordinates. They have no real musical labels:

Toy inputVector xDesired score
Piano[1, 1, 0]1
Guitar[1, −1, 0]0
Flute[0, 0, 1]0

Notice that piano and guitar share a positive first coordinate. A single coordinate does not identify the instrument; the combination matters.

6Turn the desired behavior into equations

For our three-coordinate toy, call the unknown gate coefficients a, b and c. The requested responses give three equations:

a + b = 1 piano response
a − b = 0 guitar response
c = 0 flute response

The second equation says a = b. The first then says 2a = 1. So g = [0.5, 0.5, 0].

Check it: piano gives 0.5 + 0.5 = 1. Guitar gives 0.5 − 0.5 = 0. Flute gives 0. The coefficients were calculated directly from the behavior we requested.

The real gate uses the same idea, with 768 unknown coefficients and calibration examples from the frozen encoder.

Real calibration dataHow it is used
432 text encodingsNine instruments × 12 descriptions × four style-mapper seeds
Nine synthetic audio referencesThree piano examples and six controls

These produce 16 required responses: the piano text mean and piano audio mean must score one; eight other instrument means and six individual audio controls must score zero.

Ag=b
A holds the 16 constraint vectors as rows. b holds the requested responses.

The requirements apply to those particular calibration vectors. They do not promise that every individual piano example, or every new recording, will produce exactly one.

7Choose a gate that behaves consistently

The toy had three unknowns and three equations. The real problem has 768 unknowns and only 16 required responses. Many possible gates can satisfy the constraints.

Some would behave badly. A piano mean could score one even if individual piano descriptions receive wildly different scores whose average happens to be one.

We therefore penalize variation across the piano examples, nonzero responses to text controls, and very large coefficients. The audio controls are protected by the explicit constraints. A matrix called Q packages these quadratic penalties.

mingg⊤Qgsubject toAg=b
Choose the lowest-penalty gate that satisfies the requested responses.
Read the full gate formula, one part at a time
g=Q−1A⊤(AQ−1A⊤)+b
SymbolMeaning
A · 16 × 768One calibration constraint per row
b · 16 valuesThe two desired ones and fourteen zeros
Q · 768 × 768The cost of unstable responses and large coefficients
g · 768 valuesThe gate we are solving for
⁻¹A matrix inverse in the mathematical expression; the code uses a linear solve
⁺A pseudoinverse, a generalized inverse that can handle redundant constraints

This is the direct solution to the constrained minimization above. It is not a second model or another training procedure.

The implementation first solves QB = Aᵀ. It then computes g = B(AB)⁺b. This is the same formula without explicitly constructing Q’s inverse. It checks that Ag is within numerical tolerance of b, and rejects inconsistent or ill-conditioned calibration.

Q includes centered piano-text variation, centered piano-audio variation, squared responses to text controls, and a scaled identity term that regularizes the coefficients.

You do not need to memorize the inverse expression to explain the method. The important point is the problem it solves: meet the stated responses while discouraging unstable behavior.

This is fitting from calibration data. Python calculates the coefficients through linear algebra; it does not repeatedly backpropagate through Magenta or take optimizer steps. The encoder and generator remain frozen while the examples are collected.

8Store “when” and “what” in one matrix

We now have the input coefficients g and output direction d. For every input position i and output position j, set the weight change to:

ΔWij=αgidj
One gate coefficient × one direction coefficient × the overall strength.

The outer product gdᵀ makes the whole table at once. A 768 × 1 column multiplied by a 1 × 1,024 row produces a 768 × 1,024 matrix, exactly the shape we need.

W′=W+αgd⊤
The edited adapter is the original adapter plus this coordinated table of changes.

We are not choosing 786,432 independent fixes. The update is built from one 768-number gate and one 1,024-number direction. Every input’s additional output lies along the same direction d, with a different multiplier. This is why the update is rank one, before FP32 rounding.

A complete edit you can inspect

3 inputs × 2 outputs
Invented teaching model

This toy uses the gate we solved above: g = [0.5, 0.5, 0]. Its piano conditioning is [2, 1], flute conditioning is [1, 3], and therefore d = [−1, 2].

Original W fixed

Change ΔW αgdᵀ

Edited W′ W + ΔW

Hover or select a changed weight to inspect its calculation. Rose: decrease. Blue: increase.

Gate response · xᵀg
Added output · αs(x)dᵀ
Original conditioning · xᵀW
Edited conditioning · xᵀW′

Two calculations, the same answer

Changing x changes the response, not the matrices. Changing α changes the saved matrix. The real patch fixes α at 1.4; this control is only for learning.

9The ordinary multiplication does it all

Look at just one output value, j. The extra contribution from the edited weights is the sum of each input multiplied by its corresponding weight change:

Δyj=∑ixiΔWij=∑ixiαgidj

For this output, α and dj are fixed. Pull them outside the sum. What remains is exactly the dot product that defines the gate:

Δyj=αdj(∑ixigi)=αdjs(x)

For all outputs together, that is the second README equation:

x⊤W′=x⊤W+α(x⊤g)d⊤
Original conditioning + strength × gate response × flute shift.

The left side is what the model actually runs: multiply the input by the saved adapter. The right side explains the effect. They are two ways of writing the same calculation.

There is no separate “detect piano, then add flute” program during generation. Multiplication by the edited weights already performs that operation.

10Strength is chosen. Selectivity is imperfect.

The main patch uses α = 1.4, chosen through development comparisons and listening. It is an experimental choice, not an optimum proved by the equations.

Gate responseAdded shift at α = 1.4Effect
11.4dFull intended strength
00No added shift
0.50.7dA partial shift
−0.5−0.7dThe direction reverses

These scores illustrate the calculation; they are not measurements of the demo clips. A linear gate can produce values below zero or above one. There is no automatic clamp.

A recording containing piano and strings arrives as a combined style representation. The gate does not isolate the piano track. An unfamiliar combination can receive too little change, the wrong change, or a negative response.

In the actual evaluations, some mixed arrangements failed. A piano-and-strings reference produced a negative gate response, reversing the intended change. The newer experimental patch adds four zero-response constraints, but its gate is still linear and unbounded.

Two blind quality comparisons against simply prompting the original model for flute split 1–1, with one listener. The project demonstrates selected audible changes saved in the weights; it does not establish a quality advantage over prompting or reliable preservation of every other instrument.

11Save the new rule, then use the model normally

Keep the editing stage separate from generation. Direction and gate are calculated while building the patch. During generation, their combined effect is already in W′.

Building the patch

  1. Run frozen encoders on calibration examples.
  2. Calculate the output direction d.
  3. Solve for the gate g.
  4. Choose the strength through comparisons.
  5. Calculate W′ and save it.

Every new request

  1. Encode the prompt or reference.
  2. Multiply its style vector by W′.
  3. Generate and decode the music.

No recalibration, hidden flute prompt or activation hook.

The patch stores the complete replacement matrix. Its size follows directly from its shape and number format:

768 × 1,024 = 786,432 values
786,432 × 4 bytes per FP32 value ≈ 3.15 MB, plus a small file header.

The full checkpoint contains 2,459,163,592 parameters, including the codec. The replaced matrix is about 0.032% of that total. The other 617 tensors are unchanged.

That percentage counts affected values; it is not the size of each numerical change. Within the adapter, the update’s Frobenius norm is 5.026% of the original matrix’s norm. The small global fraction can matter because this matrix carries style information into the generator.

VerificationWhat it establishes
Compare checkpoint tensorsOnly the intended matrix changed.
Rebuild from saved calibration tokensThe recalculated patch has the same SHA-256 hash.
Reload in a fresh processOne development recording reproduced byte for byte on the tested runtime.
Matched prompts, seeds and settingsBefore-and-after generations isolate the checkpoint change under those settings.

These checks establish properties of the saved artifact and tested runs. Listening comparisons are still needed to judge instrument identity, sound quality and accompaniment.

Explain it to someone else

“Google’s Magenta already knows how to generate piano and flute. We used examples to calculate a shift between their internal style representations, and a score that makes this shift respond mainly to piano inputs. We combined those calculations into one replacement matrix and saved it into the model. The same piano input can then produce flute through the original generation code, without backpropagation or an extra runtime instruction. Keeping the change selective on unfamiliar or mixed-instrument inputs is the difficult part.”

Where the WSP part comes in

This is a human-guided, tool-assisted weight-space programming experiment. GPT-6 Astra designed and implemented the main edit using Python and frozen encoder examples. I set the goal and supplied listening feedback.

The mechanism is a calibration-based rank-one update using established linear algebra. Magenta’s flute-generating ability already existed; the edit changes when that ability is invoked. It does not imply that someone identified individual “piano weights” by looking at raw numbers.

Back to the beginning