# Experiment design

## Task and data

Images use integer 8-bit RGB values, exactly matching the canvas display. They contain one light upright or horizontal crossbar on a blue or orange background. Shape orientation is the binary label. Position, length and low-amplitude pixel texture vary deterministically. The original training environment makes background and label perfectly correlated; this is deliberately extreme so the failure is visible and reproducible.

The generator creates 128 training specimens and 192 held-out specimens. Train/test/probe sample seeds occupy separate high-bit domains, preventing seed identity overlap. A within-split set prevents repeated sample seeds. Tests also check no exact pixel duplicate crosses the default train/test split. Geometric parameters can repeat because the shape family is intentionally small; different sample seeds and textures do **not** imply entirely novel shape families.

Held-out labels are balanced. The shifted test is a paired intervention on the same 192 held-out specimens, not another independent set of 192 samples. The original and shifted evaluations therefore should not be pooled as 384 independent observations. Each shape × backdrop group has 96 specimens.

## Intervention

The counterfactual re-renders only pixels outside the original foreground mask using the other backdrop color. It preserves every foreground RGB value, the shape, position, length, texture realization and ground-truth label. It is not a second randomly drawn image.

Generator metadata is used to construct the controlled intervention and score predictions. **Prediction receives only an RGB pixel array.** Feature extraction does not accept an event ID, label, backdrop label or geometry parameter. Training uses the ordinary supervised target label alongside pixel features; that is supervision, not a hidden inference shortcut.

## Model and objective

For each of 768 RGB components:

```text
x[j] = (pixel[j] / 255 - 0.5) / sqrt(768)
p(crossbar | x) = sigmoid(dot(w, x) + b)
```

The classifier has 768 learned weights plus one bias. It predicts crossbar for probability ≥ 0.5 and upright otherwise. It is linear in raw pixels; no shape rule or manually coded output is used at prediction time.

Both models initialize all weights/bias to zero and perform 120 full-batch gradient-descent updates with learning rate 4. The objective is mean binary cross-entropy plus `0.008 * ||w||² / 2`; the bias is not regularized. The loss curve records the objective before each actual update. Numerical finite-difference tests verify gradient components, including regularization and bias.

The baseline fits 128 correlated views. The balanced model starts again from zero and fits 256 views: both backgrounds for each **training** specimen, with unchanged labels. It does not fine-tune on the held-out set, inspect test labels during fitting or change optimizer hyperparameters to force a result. The held-out set is evaluated after fitting.

## Audits

- Original-environment accuracy: held-out examples with the same background/label relationship as baseline training.
- Shifted accuracy: the same examples with backgrounds reversed.
- Worst-group accuracy: minimum accuracy across the four shape × backdrop cells from those two environments.
- Prediction-flip rate: fraction of paired specimens whose predicted class changes after only the backdrop changes.

Per-group counts are shown. These are finite-sample descriptive measures, not confidence intervals or certified robustness bounds. Worst-group accuracy here concerns synthetic shape/background cells, not demographic fairness groups.

## Uncertainty and scope

The sigmoid output is displayed as an **uncalibrated model score**, not a measured probability that the prediction is correct. After balancing, default scores are near 53–54%, so the model has a small margin despite perfect accuracy on the chosen simple fixtures. It has not been calibrated or tested on rotated shapes, severe noise, new palettes, occlusions, arbitrary photographs or natural distribution shifts.

The two-class task assumes every image is an upright or crossbar. There is no unknown-class detector. The point is to make a controlled shortcut visible, not to claim that balancing one feature makes a model generally safe. The method is pedagogical, not a new ML algorithm or an evaluation of any external model.

## Reproducibility and privacy

The `shiftglass/1` recipe fixes seed, sample counts, model parameters, data family and training modes. Import rejects unsupported values before resetting the experiment. It ignores stored audit results and reconstructs both models. Tests reproduce weights exactly within the same JavaScript runtime; different engines may differ in floating-point details. The verifier prints weight hashes to identify an exact local run.

All computation stays in the browser or local Node process. Export is a local JSON download. No model weights or images are fetched. No accounts, telemetry, uploads, payments or external inference are part of the application.