marola-bench: a proposal for a WeatherBench for Brazil

What the bench is

“The bench are the score series performed by any Forecaster, Registered by an Actor.”
Matheus Hoffmann, the owner

A score is one forecast compared with one observation. A series is those scores in order of time, for one forecaster, one point, one variable and one lead. The bench is all the series, and it is the final goal: the points, the actors and the cloud exist to make it.

Three things the public benchmarks do not have

the paper, sections 1 and 2

WeatherBench, WeatherBench 2 and Operational WeatherBench made weather models comparable. marola-bench keeps their method: one scorer, a truth that is stated one time, fixed leads and regions, a rule written before the scores. It changes three things.

A region that is Brazil
The operational scoreboard offers the globe, the tropics, the two hemispheres, North America and Europe.
A score for waves
Its variables are of the atmosphere. marola forecasts the sea.
An instrument as the truth
There, a model is scored against its own analysis. Here, against what a buoy or a station measured.

7 hard rules

the research's page

The owner's rules. They are not open to an agent's choice. Five of them change the design that was accepted the same day, and the paper says where; a new design document must settle the rest before code is written.

marola-bench is implemented in Apache Pekko, in Scala 3.
marola-bench is implemented on GCP, and developed with Claude Code, with an SDD (spec-driven development) framework.
Its infrastructure as code is defined with Besom.
marola-bench lives here, in marola-research/marola-bench. The first rendered paper is here.
The deployment of the Pekko cluster is cheap and reliable. Both are hard requirements, and so is its infrastructure on GCP.
marola-bench is all rights reserved, and the credit belongs to Matheus Hoffmann. No sharing outside marola-dev and ouroboric-dev without his acknowledgement. Whoever uses it cites it.
Ponytail 5 is adopted as the coding assistant, a Claude Code plugin (the owner, 14 October 2026).

How it works

the paper, section 4

A person adds a point. A buoy or a station, through an API. By hand, because a point has an instrument, a quality flag and a licence, and somebody must read the terms.
The point gets an actor. One small program for one point. It knows when the next observation is due.
The actor asks the forecasters. The providers that publish a model today, and marola's own forecaster later. It asks all of them the same way.
The actor registers a score. One forecast against one observation. A score that was registered is not edited.
The scores are the series, and the series are the bench. marola's forecaster is in it as one forecaster among the others.

Cheap, and reliable

the paper, section 4.4

Reliability is what makes it cheap.

Cheap capacity on a cloud is the kind that can be taken away. A cluster that survives the loss of any node can use it. So the two requirements are one piece of work: first survive the loss of a node, then buy the cheap nodes. The four properties of a reactive system are used for both.

Property	Cheap	Reliable
Message driven	An actor that waits uses no processor.	One failed point or provider does not stop the others.
Resilient	A node may be lost, so capacity that can be taken away is usable.	An actor's state is its stored events: no score is lost with a node.
Elastic	An idle actor is stopped. Memory follows the work, not the number of points.	More points do not need a larger design.
Responsive	No capacity is kept for a peak.	An answer that does not come in time is a row that says missing, never a guess.

These are proposals for the next design document. Nothing has been priced, and no cluster exists.

Where it stands

the journal

doneThe proposaldraft, v0 (a proposal, not peer reviewed; no score has been computed).
doneThe references35 of them, each read against a passage of its own text.
doneThe accepted designMIP-0091, in marola's main repository.
openA design for the hard rulesHow Pekko, GCP and Besom meet the accepted design.
noneCodeNone yet.
noneObservation pointsNone yet.
noneScoresNone yet.

Read

the source

Public weather benchmarks made forecast models comparable, but for a person on a Brazilian beach three things are missing in them: a region that is Brazil, a score for ocean waves, and an instrument as the truth. This paper proposes marola-bench, which keeps the method of WeatherBench, WeatherBench 2 and Operational WeatherBench and changes those three things. The bench is defined as the set of score series: the scores of any forecaster against the observations of a point, in order of time. Observation points are added by hand, and one actor for each point asks the forecasters and registers the scores. The paper states seven hard rules of the owner: five for the implementation (Apache Pekko and Scala 3, GCP, Besom, a home in this repository, a cheap and reliable cluster), one for its terms (all rights reserved) and one for its coding assistant (Ponytail 5, a Claude Code plugin). It says where they change the accepted design, MIP-0091. It reports no result: no forecast has been scored, and nothing has been built.

Summary
What is copied, and from whom
The proposal
The system: the bench, the points and the actors
From a scoreboard to one forecast
Conservative, and safe outside the forecaster
Why this is open
The order
What is not known
Where the thesis comes from

The preview does not show on every phone. Open the PDF or download it.

Notes

the slip box

One idea in one note, linked to other notes, each with how far it was checked. The paper is written from them.

marola-bench is marola-experiment 001fleetingverified
The design is narrower than the first notepermanentverified
Zero spend meets a paid cloudpermanentopen question
Ten buoys, three analysedliteraturepartial
Operational WeatherBench, read againliteratureverified
Reliability is what makes it cheappermanentpartial
An actor for each observation pointpermanentverified
Seven hard rulespermanentverified
The bench is the score seriespermanentverified

Journal

every entry

One entry for each challenge met: what was hard, the options, the decision, the result and the lesson. What went wrong is an entry too.

A seventh rule, and a test that counted six2026-10-14closed
A freedom that a check can see2026-10-14closed
The first record came out open2026-10-13open
Five rules that change an accepted design2026-10-13open
Five rules in one place and six in another2026-10-13closed
A paper written before its design was found2026-10-13closed
A paper that argues for openness, closed2026-10-13open
A DOI in five places and missing from three2026-10-13closed

References

how each one was checked

No reference is cited before it is fetched. For each one, a passage of its own text that supports the claim was read, and the log quotes it and says how much of the source was read.

35references
6read as full text
9read as an abstract
20read as a page or a record

Checks

all runs

A change to the paper is a proposal first. Machines read it, then a person decides. The same checks run on the proposal and on the main branch, and each badge is the state of that check now.

the paper builds, every reference has its row and is cited, and the PDF is the one its text makes.
structure, plain words and links to files, in every Markdown file; the notes keep their rules.
every web link answers.
this page builds from the research's files and its tests pass.