Articles

Articles by Alexander Chernov, first published on LinkedIn and reprinted here. Each page links to the original, which remains the canonical version.

© 2026 Alexander Chernov. All rights reserved. Each article was first published on LinkedIn; the linked original remains the canonical version.

At 100,000 Devices, Nothing Fails at the Average

By Alexander Chernov. First published on LinkedIn, 2026-09-01. Read the original.

Part 2 of two: what happens when things move.

Part 1 measured where load lands in a 100,000-device estate, and the result was blunt. Hash 100,000 equipment ids through Kafka’s murmur2 keyed-partitioning calculation, weight each key by its device’s rate, give one instrument a fault, and going from 16 partitions to 1024 improves the cluster average by exactly 64× while improving the actual bottleneck by 1.25×. The 64 is just 1024/16 — the mean is total load over partition count, so it reports the change you made to the denominator and nothing about the cluster.

That was a system standing still. This part is about the system moving: a site that reconnects with twenty minutes of backlog, a fleet that all reports on the same second, a pipeline that ships a schema change, and a certificate that expires on a Sunday. Every one of them is a failure that an average cannot see, for the same reason the average could not see the hot partition.

Two of the three measurements are here; the harness and the browser calculator are the same ones, at doytsujin.github.io/ok-kafka-estate-calc. As in Part 1, everything that is not measured is arithmetic from stated assumptions and is marked as such.

Recovery, not average traffic, is the design point

A manufacturing site with 12,000 devices loses its upstream network for twenty minutes. The devices keep operating and buffer locally. Then connectivity returns and every device starts replaying at once. At 1 Hz that is 14.4 million buffered events arriving at a platform that was sized for 12,000 a second, and the pressure propagates through every layer in turn.

This one has a closed form, and it is worth knowing. If each device replays at k times its normal rate while still producing new data, the backlog drains at (k-1) × normal and clears in outage / (k - 1). Simulation and closed form agree to the timestep.

At k = 2, recovery time equals outage time. Twice normal capacity buys a recovery exactly as long as the outage that caused it. Comfortable-sounding headroom is not comfortable: at 1.25× a twenty-minute outage takes eighty minutes to clear, and for all eighty of them the platform is running at its ceiling — which is exactly when the second failure arrives.

So the capacity question is not what the average ingestion rate is. A system that comfortably handles average traffic can fail precisely when it is trying to recover from the previous failure, and that is how cascading failures begin.

The schedule is worth a factor of 56

Now suppose 100,000 devices are configured to report once per minute. The average rate is about 1,667 events/sec. Nothing frightening — unless every device reports at 12:00:00, 12:01:00, 12:02:00.

I ran all 100,000 devices through three reporting schedules and measured the peak events landing in any one-second window.

Same devices, same volume of data, same retention. The schedule alone is worth a factor of 56 in peak load, and the middle row is the one worth staring at: devices that came back in a handful of restart waves — a site power event, a fleet firmware push — still cost 5×, because partial synchronisation is the normal state of any estate that has ever been restarted in batches.

The problem is not insufficient Kafka tuning. It is a thundering herd, and the fix is jitter applied before the event ever reaches Kafka.

The topology is code, or it is a liability

Once Kafka is production-critical, topics, schemas, ACLs, brokers, connectors, SLOs, upgrades and incident response are software-delivery concerns rather than messaging concerns. Many of the hardest failures in that infrastructure begin with an apparently routine change somebody shipped.

That puts Kafka across the whole lifecycle. Application changes introduce new producers, consumers, topics, schemas and partitioning behaviour, and CI should validate them before rollout rather than after. Clusters, topics, quotas, ACLs, connectors, service accounts, retention policies and monitoring are declarative artefacts and belong in Terraform or GitOps alongside everything else. Observability means consumer lag, request latency, ISR health, rebalances, broker saturation and end-to-end event latency, at the granularity the drill-down above demands. Replication, acks, min.insync.replicas, failure-domain placement, capacity headroom, retries, DLQs, idempotence and cross-region replication are reliability settings that live in version control. And because Kafka sits in the middle of most incident chains, runbooks have to distinguish producer, broker, consumer, schema, network and downstream failures rather than paging on “Kafka”.

Upgrades deserve their own sentence, because protocol and client-version compatibility is the classic way a routine change becomes an outage. Progressive rollout, compatibility testing and a rehearsed rollback are not optional at this scale.

Nothing exotic. It is ordinary DevOps applied to a component that often escapes it because it is filed under “messaging”.

Security controls are part of the availability model

Kafka in a regulated environment carries data that has an access model, and that adds a set of questions the throughput conversation never reaches. Who can produce, who can consume, and to which topics? Can one tenant read another’s data? Are credentials rotated? Is traffic encrypted in transit and at rest? Can sensitive fields leave their permitted boundary? Can an administrator alter retention or ACLs without leaving a trace?

The controls are well understood: TLS, SASL or OAuth rather than shared passwords, topic- and cluster-level ACLs, per-service identities, secrets in a real secrets manager, certificate and credential rotation, network segmentation, schema and payload validation, quotas, audit logging, retention controls, and scanning of both Connect plugin images and the dependencies of producer and consumer applications. Provisioned the same way as everything else.

Here is the part that matters for reliability, and it is the reason this section is in a throughput article at all. Security controls cause outages, and they cause the quiet kind.

Consumer lag looks normal. It looks normal because lag measures the distance between what was written and what was read, and nothing was written. The health metric is arithmetically incapable of seeing the failure, in the same way the cluster average is incapable of seeing the hot partition. A producer-side certificate expiry presents as a quiet system.

The other two chains in that figure are noisier but no less real. A rollout that removes a consumer’s READ permission turns it into a retry loop that never drains, and a compromised or buggy producer publishing a hundred times its normal volume becomes broker pressure and then cluster degradation.

That third one is the hot-partition measurement again, arriving through the security door instead of the firmware door — and the quota that stops it is a security control doing reliability work. Authentication, authorisation, quotas, certificates and network policy are part of the availability model, not adjacent to it. Teams that separate the two end up with a security calendar that schedules outages.

The difficult questions are about failure

At 100,000-device scale I am not very interested in whether Kafka can process 100,000 messages per second. It is a useful benchmark and it says surprisingly little about whether the system is reliable. I would rather ask:

• What happens when a site reconnects and replays thirty minutes of buffered telemetry?
• What happens when one consumer falls hundreds of millions of events behind?
• Can an availability-zone failure occur without losing the manufacturing event stream?
• What happens if a firmware defect causes 20,000 devices to publish a hundred times more frequently?
• Can we distinguish a duplicate measurement from a legitimate repeated measurement?
• What happens when an event arrives forty-five minutes late?
• Which certificate expires next, and what stops when it does?

And, in a regulated environment, the one that matters most: can we reconstruct the exact state of a batch at a particular point in time?

That question leaves throughput behind entirely. It touches ordering, event time versus processing time, replay, idempotency, schema evolution, provenance, retention, auditability and state reconstruction. Those are distributed-systems questions, they are reliability-engineering questions, and increasingly they are delivery-pipeline questions, because the answer depends on what a deployment three months ago did to a schema.

Reliability starts where the benchmark ends

At sufficient scale, backpressure, consumer lag, partition skew, ISR degradation, retry amplification, gray failures, recovery bursts, capacity headroom, late-arriving events and failure-domain isolation stop being vocabulary. They become architectural properties of the system, and each one is a thing you either designed for or did not.

Three of them now have numbers attached across the two parts, and the numbers point the same way each time. The cluster average improves by exactly the factor you grew the denominator by, while the bottleneck does not move. Lag reads normal while nothing is being published. A schedule nobody chose costs a factor of 56. In every case the reassuring measurement and the actual failure are looking at different things, and the reassuring one is the one on the wall.

If you want the three measurements against your own estate rather than mine, the calculator is at doytsujin.github.io/ok-kafka-estate-calc. It is deliberately not a broker benchmark and has no output that answers whether your cluster can cope — that question needs your cluster. What it will tell you is where the load actually lands, what your reporting schedule costs, and how much headroom your recovery needs.

The challenge is not moving a very large number of messages. It is that when devices, networks, brokers, consumers, sites, certificates and downstream systems behave in unexpected ways — and they will — the platform has to absorb the disturbance, preserve the meaning of the data, recover predictably, and tell you exactly what happened.

Throughput is where that conversation starts, not where it ends.

© 2026 Alexander Chernov. All rights reserved. First published on LinkedIn, which remains the canonical version; this page is a reprint by the author.

At 100,000 Devices, Throughput Is the Easy Part

By Alexander Chernov. First published on LinkedIn, 2026-08-31. Read the original.

Part 1 of two: where the load actually lands.

When someone says a hundred thousand pieces of equipment are connected to a streaming platform, the first question is almost always whether Kafka can handle it. It is a reasonable question, and it is close to the least interesting one available. At that scale the hard engineering is in bursts, backpressure, partition skew, replay, late data, recovery behaviour, failure domains, and reconstructing what actually happened — and a throughput benchmark answers none of it.

This gets more interesting in biopharma, where a “device” might be an environmental sensor, a bioreactor, a chromatography skid, a laboratory instrument, a PAT system, a sequencer, or an imaging platform. One device rarely means one signal.

I have not benchmarked a cluster of this size and this article does not claim to. What I did instead was measure the things that do not need a broker: the partitioner, the reporting schedule, and the arithmetic of recovery. Those are exactly computable over a real key space, and they run in the browser at doytsujin.github.io/ok-kafka-estate-calc if you want to put your own estate in — no cluster, nothing uploaded.

This part is about where the load lands: the arithmetic of the estate, what a key actually is, and the one measurement that decides how a partitioned log behaves under it. Part 2 is about what happens when things move — recovery, reporting schedules, deployments and certificates. Everything here that is not measured — the estate, the device rates, the storage figures — is arithmetic from stated assumptions, and I have marked it as such.

Start with the arithmetic

A useful first approximation is events/sec = devices × events/device/sec.

With 100,000 devices, a small change in reporting frequency changes the architecture dramatically.

Two orders of magnitude in device rate is four orders of magnitude of difference in what you have to build, and those are only raw payload numbers. They exclude replication, protocol overhead, retries, indexes, downstream processing, derived events, and consumer traffic.

One device is not one measurement

A bioreactor or chromatography skid might expose temperature, pH, dissolved oxygen, agitation, pressure, flow, conductivity, UV, valve states, pump speeds, alarms, recipe state and batch state — potentially dozens or hundreds of tags. A naive architecture therefore turns 100,000 devices × 100 measurements × 1 sample/sec into ten million measurements per second. That does not mean ten million Kafka messages per second. The representation is a design decision, and it is made before anything reaches the broker.

Instead of publishing every tag independently, an edge or acquisition layer can produce one contextualised observation.

Where sampling semantics allow it, a hundred individual measurements become one semantically useful equipment observation. That is not an optimisation. It changes the shape of the entire distributed system, because the unit that gets ordered, partitioned, replayed and audited is now an equipment state rather than a scalar.

A 100K-device estate is heterogeneous

I would not model a large biopharma environment as 100,000 identical producers. An illustrative estate looks more like the one below, and it is the one the harness runs on.

That averages around 78,000 events/sec, or roughly 6.7 TB/day of raw event data at 1 KB per event. With replication factor 3 the brokers are writing on the order of 20 TB/day across replicas. Keep seven days online and the storage footprint is substantial before you add safety margin, segment overhead, tiered storage or consumer traffic.

Note where the volume lives. Five per cent of the devices produce nearly two thirds of the events. The estate is not one workload with one number attached to it, and treating it as one is how the interesting failures get averaged away.

A “100,000-device Kafka architecture” is therefore not a workload specification. Device count tells you the size of the estate. The distribution of event rates tells you what system you actually have to build.

This is unquestionably a serious streaming workload. Average throughput is still not the part I would worry about most.

Partitioning solves ordering, and creates its own questions

For equipment telemetry a natural Kafka key is equipment_id, which keeps every observation from BR-001 in the order it was produced. That does not mean one partition per piece of equipment: 100,000 devices is not 100,000 partitions, and a cluster configured that way would collapse under its own metadata.

Instead, hash(equipment_id) → partition distributes large numbers of equipment streams across a manageable partition set while preserving ordering per equipment identifier. The precise partition count depends on throughput, workload isolation, consumer parallelism, recovery requirements and operational constraints.

Hashing does not eliminate the next problem, which is skew — and this is the one I most wanted a number for.

The hot-equipment problem, measured

I put all 100,000 equipment ids through Kafka’s murmur2 keyed-partitioning calculation and weighted each key by its modelled event rate. The exact transcription, seed and modulo are in the calculator’s source at github.com/doytsujin/ok-kafka-estate-calc. Then I gave one analytical instrument a firmware defect and let it publish at 20,000 events/sec instead of 10.

The first column is the hash doing its job. Key counts stay within five per cent of uniform, which is the result everyone expects and quietly stops checking after.

The rest of the table is the actual behaviour. Even with no hot device, load skew is already worse than key skew and grows with partition count, because devices do not all publish at the same rate and the hash cannot know that. Add one hot device and the ratio goes to 13.89× at 64 partitions and 209.56× at 1024.

Read the last two columns together, because that is the finding. Going from 16 partitions to 1024 improves the cluster average by exactly 64x — from 6.12 MB/s to 0.096 MB/s — and improves the actual bottleneck by a factor of 1.25, from 25.1 MB/s to 20.1 MB/s.

The 64 is not a measurement. It is 1024/16. The mean is total load over partition count, the total load did not change, so the average was always going to improve by exactly the factor you multiplied the denominator by. It is arithmetically incapable of reporting anything else, and it will do this on your cluster too, whatever is wrong with it. Every partition you add makes the dashboard look better and the problem stay the same. A single key is an unsplittable unit of load, and no partition count divides it.

I ran the whole thing again under an independent SHA-256 partitioner as a control. The numbers land within noise of murmur2 throughout, which is the point: this is a property of the workload, not of a hash function, and it will not be tuned away.

That is also why aggregate dashboards are insufficient rather than merely coarse. At this scale I want to be able to move down through the levels.

The failure lives several layers below the average, and the average gets more reassuring the further you drill away from it.

Kafka probably should not carry the rawest scientific signal

Suppose an instrument produces a 10 kHz waveform, a large microscopy image, a chromatogram, sequencing output or a large spectrum. You can push these payloads through Kafka. For an enterprise biopharma architecture it is usually the wrong abstraction.

The object store holds the large payload. Kafka carries an event describing what happened, and that event is a few hundred bytes rather than a few hundred megabytes.

The event stream now carries meaning and state while object storage carries bulk scientific data. Kafka does not have to become a scientific file system in order to be central to the architecture.

Three data planes

For a large biopharma environment I find it useful to separate three classes of information.

Operational events — EquipmentStarted, BatchStarted, ValveOpened, AlarmRaised, SampleCollected, RunCompleted, QualityCheckFailed. Compact, semantically rich, and a description of things that happened.

Telemetry — temperature, pressure, pH, flow, dissolved oxygen, RPM, environmental measurements. Higher-volume continuous information, and the plane most likely to be aggregated or downsampled before it reaches the enterprise streaming layer.

Bulk scientific data — spectra, images, chromatograms, sequencing data, microscopy, raw instrument files. These belong in object storage or another system optimised for large scientific objects, with Kafka carrying the event announcing that the object exists, where it is, what generated it, and how it relates to the surrounding process.

The three planes then work together instead of forcing every form of data through the same transport abstraction.

Kafka becomes the nervous system, not the entire body

Put the three planes and the estate together and the architecture takes a definite shape.

At that point Kafka is not simply a message broker. It is closer to a nervous system carrying operational state across the estate — which means it stops being an application dependency and becomes production infrastructure, with everything that implies about how changes to it are made.

What this part settles, and what it does not

One number is now firm, and it is not the one on the dashboard. Load skew is a property of the workload and not of the hash: a single key is an unsplittable unit of load, no partition count divides it, and every partition you add makes the average look better while the bottleneck stays exactly where it was. The cluster mean is arithmetically incapable of reporting anything else.

That is a statement about a system standing still. It says where the load sits when the estate is running normally, which is the easy case. It says nothing about what happens when twelve thousand devices reconnect at once, when every device in the fleet reports on the same second, when a deployment changes a schema, or when a certificate expires quietly on a Sunday.

Those are the cases where estates actually fail, and none of them shows up in an average either. That is Part 2.

© 2026 Alexander Chernov. All rights reserved. First published on LinkedIn, which remains the canonical version; this page is a reprint by the author.

From Governed Data to Governed Motion: A Control Plane for Biomedical Robots

By Alexander Chernov. First published on LinkedIn, 2026-08-25. Read the original.

For the last few years the hard problem in enterprise AI has been governing data — making sprawling, heterogeneous data usable, traceable, and trustworthy enough to put a model on top of it. I’ve written before about agentic datasets: data that carries its own descriptor, policy, provenance, and the ability to refuse to be used in ways its policy forbids.

But in a laboratory, the AI’s “output” is not a chart or a summary. It is a robot arm moving a plate of live cells through a sterile workspace. When the consumer of a governed decision is a physical manipulator, the same control-plane ideas stop being about tidiness and become about safety. The frontier moves from governing data to governing motion.

In the physical world, a bad action is not free

In many analytical software systems, a wrong output can be caught, recomputed, or rolled back before it touches the world. Robotic actuation is different: a wrong action moves a real manipulator — possibly into a locked sterile zone, through an occupied fixture, or while an operator’s hand is in the workspace — and once motion begins, the consequences can be immediate and irreversible. “Check the policy, then probably act” is not good enough, because the gap between the check and the act is exactly where accidents live.

So the requirement inverts. It is not enough to log that a rule was violated. The architecture has to make a refused intent structurally incapable of reaching the normal actuation path. Refusal has to be structural, not a runtime afterthought. That is a claim about the software architecture and not about the physical system: a controller bug, a bypass path, a hardware fault, or somebody jogging the arm by hand all sit outside it, which is exactly why the layers below still matter.

A supervisory control plane above the motion stack

The shape that delivers this is a supervisory layer that sits above the low-level controllers and planners — it does not replace them. Each robot is modeled as a long-lived agent with an explicit lifecycle: it reasons, then plans, then executes, then publishes results, then returns to idle.

The trick is where policy is evaluated. Policies — sterility, human safety, chain of custody, workspace occupancy — are checked in a dedicated reasoning phase, before any planner or driver is touched. A denial returns the agent to idle without ever entering planning or execution. Because the only side-effectful steps live past that gate, “no actuation on a denied intent” becomes a property of the state machine itself, not a promise made by careful code.

This plane does not replace functional safety. Emergency stops, hardware interlocks, safety-rated controllers, collision limits, and workspace monitoring remain the final protective layer, and nothing here weakens them. The supervisory plane operates earlier and answers a different question. Conventional safety asks, “how do we stop an unsafe motion?” The supervisory plane asks, “why was this motion authorized to begin?” Two different predicates: governance decides whether an operation is authorized, functional safety decides whether the resulting motion stays physically safe, and neither one answers the other’s question. Three layers, three jobs: the supervisory plane governs task admission, intent, provenance, and accountability; the safety controller enforces hard real-time physical limits; the motion stack does planning and actuation.

Three properties then follow from the architecture — enforced by construction, provided its state and side-effect boundaries are respected:

• Refusal. Unsafe intents are rejected before the robot can move, and each refusal record says which policy was violated and why.
• Traceability. Every decision, plan, execution event, and telemetry sample carries the originating intent’s trace identifier, so the full causal chain can be reconstructed after the fact.
• Observability. An operator can watch, live, what each agent is doing, why it decided what it decided, and where a task was interrupted — without stopping the running system.

What a governed intent carries

The word intent is doing a lot of work above, so it’s worth making concrete. A governed intent is not a natural-language command or a bare motion request. It is a structured runtime artifact that carries enough context for policy to be evaluated before anything moves: the identity of the requesting workflow, the sample and batch in play, the required sterility and workspace state, the permitted equipment, the operator’s authorization, time constraints, the applicable policy set, and a trace identifier.

intent: transfer_plate
source: incubator_2
destination: reader_1
sample:
  batch: batch_24
  plate: plate_07
required_state:
  sterility: sterile
  workspace: unoccupied
requested_by:
  workflow: assay_run_12
  operator_role: automation_operator
policy_set: biosafety_v3
trace_id: tr-8f31

The policy engine therefore evaluates not just what the robot is asked to do, but under which conditions it is allowed to do it. This is the same move as an agentic dataset’s descriptor — identity, provenance, policy, and chain of custody travelling with the thing being governed — applied to a motion request instead of a data asset.

An intent is also not a trajectory. It states which operation is authorized and under what conditions; how the arm actually gets there — the joint path, the velocity profile, the collision checks — stays with the planner. Keeping those separate is what lets one policy hold across machines that move in completely different ways.

Beyond the gate: authorization can go stale

Admission control is necessary but not always sufficient. Some conditions are stable for the length of a task; others are leases that can expire mid-motion. A workspace becomes occupied, a sterile boundary is opened, an instrument enters a fault state — after the intent was authorized but before it completed. For those conditions the authorization has to stay continuously valid, or be re-checked at defined execution checkpoints. Governed motion needs both pre-actuation admission and runtime revocation: a policy-triggered suspension the moment a safety predicate stops holding.

The underlying shift is short to state and long to build. Authorization is a state, not a one-time decision. An intent has to be authorized when it starts, remain authorized while the predicates it relied on still hold, be revocable the moment they stop, and terminate through a path somebody is accountable for. A gate that answers once and never again is not governing motion; it is stamping requests.

Refusal before actuation is likewise only one failure mode. Once a task is underway, the system needs defined responses for loss of telemetry, a planner timeout, a controller disconnect, a revoked policy, or a partially completed transfer. The safe response might be to stop, hold position, retreat to a known pose, quarantine the sample, or hand off to a human — but it has to be an explicit part of the lifecycle, not exception-handling buried inside a driver. Governing how a task stops matters as much as governing whether it starts.

What the control plane must guarantee

Stated as testable properties rather than aspirations, a governed-motion system should guarantee:

• No unauthorized actuation — a denied intent cannot reach a planner, controller, or actuator.
• No orphaned motion — every physical action stays bound to an active intent, a policy decision, and a trace.
• Revocable authority — if a safety-relevant condition changes, execution can be suspended or terminated in flight.
• Deterministic recovery policy — every recognized failure condition maps to a defined recovery or escalation transition. What the architecture can guarantee is the response, not that the plant physically reaches the target state: an actuator that has failed will not retreat to a safe pose because a policy says so.
• Accountable human authority — an operator can inspect, interrupt, or assume control without stepping outside the audit trail.

That last one matters. Human oversight belongs inside the control model, not as an external escape hatch. An operator may pause, abort, or take over, but each intervention stays attributable and policy-bound. Emergency action is always available; an administrative override is a different thing, and should demand explicit authority, a rationale, and an audit record. Automation does not remove human responsibility — a good control plane makes it legible. Expressed this way, the guarantees become things you can actually check: as event-log verification rules today, and as invariants or model-checking conditions as the system hardens.

Why this matters for bionic and biomedical systems

Biomedical automation is exactly the setting where this pays off. Sterility, human-safety lockouts, sample integrity, and chain-of-custody are not optional features; they are the reason the system is allowed to run at all. A supervisory layer that makes every action explain itself — and every refusal carry its rationale into an audit trail — is what turns “we automated the lab” into an automation architecture whose decisions can be inspected, traced, and validated. The architecture does not confer approval; it produces the evidence any approval would have to rest on.

It also generalizes across hardware, and that is more than a convenience — it is where the architecture earns its keep. The same intent-level workflow — transfer this plate — runs on a Cartesian liquid handler and on a six-axis articulated arm. The boundary is the useful part: a governed intent names the operation and its conditions, an embodiment-specific planner turns that into a trajectory, and the controller actuates it. Policy attaches at the semantic layer and stops there. The kinematics differ per machine; the governance does not, so it is written once at the intent level instead of being reimplemented, slightly differently, inside every robot’s controller.

The honest next step

To be precise about what is proven and what is designed: I validated the admission invariant in simulation, verifying it directly from the recorded event log — refused intents never reached a planner or driver, and every trace reconstructed end to end. The runtime revocation, recovery, and human-authority guarantees above extend the same supervisory state-machine model, but they are architectural commitments rather than measurements, and none of them has been run on hardware. Simulation proves the design is internally consistent; it does not prove it survives contact with a real motion stack.

If you would rather see the gate than read about it, there is a small public demonstrator: a two-link arm and a discrete lifecycle running in the browser, where you can put an operator in the workspace, lock the sterile zone, and watch intents get refused with the invariant checks updating live (source). It is a demonstrator and not an experiment, and it says so on its own front page: no physics, no hardware, no sensing, and the policies, the workcell and the conditions that trigger a refusal were all written by the same hand. What you are watching is an architecture doing what it was constructed to do — useful for seeing the shape of the idea, and not evidence that a real robot is safer.

That is the work I’m on now: pushing the supervisory layer down onto a real robotics middleware as an admission gate, so that a policy denial is bound to controller activation: the controller that would drive the task is never activated, and an inactive controller claims no command interfaces, so a refused request has no path to the hardware. “No actuation on denial” is then enforced by the middleware rather than by the supervisor politely declining. Worth being exact about what that does and does not mean — it is not a matter of shutting the robot down. Whatever is holding the arm in place stays active; what never happens is the denied task acquiring a claim on the interfaces it would need to move. Runtime revocation and safe-state recovery harden on that same real stack. Transport jitter, scheduling, and real control budgets are where the interesting engineering lives.

The pitch, in one line

We learned to make data that can refuse to be misused. Move the same idea into the physical world and you get motion requests that must prove they are authorized before they become physical actions — a control plane where refusal, traceability, revocation, and recovery are properties of the architecture, not features bolted on after something goes wrong.

Part of this thinking is in a paper accepted at IEEE CBS 2026 (Cyborg and Bionic Systems), Munich, September 2026.

#AgenticAI #Robotics #BionicSystems #LabAutomation #ControlPlane #Safety #ROS2 #Bioprocessing #DataGovernance

© 2026 Alexander Chernov. All rights reserved. First published on LinkedIn, which remains the canonical version; this page is a reprint by the author.

A Game Is a Data Problem: What Reimplementing an Engine Taught Me About Silent Contract Failure

By Alexander Chernov. First published on LinkedIn, 2026-08-23. Read the original.

Nobody files a game under data engineering. Games are graphics, physics and frame budgets — a rendering discipline, shelved next to shaders and GPUs. Data work lives somewhere else entirely, in warehouses and pipelines and schemas.

I spent several weeks rebuilding a 1993 game engine so that it runs in a browser, and came away convinced that shelving is wrong. The result is playable at AgentArena.ca, which is the world I am building as a controlled environment for assessing agentic behaviour and implementation. The rendering was the part I expected to be hard. It was not the part that consumed the effort, and it produced nothing worth writing down. Almost every difficult hour went into the data — and every one of those hours taught something that transfers directly to pipelines which have never drawn a pixel.

Why “from scratch” is the whole premise

Almost every version of this game that runs in a browser is the original 1993 C source compiled through Emscripten: the same engine, re-hosted. What I built is a reimplementation written from scratch against the file formats, sharing no code with the original.

That distinction is not about bragging rights. A transpiled port inherits thirty years of accumulated handling for every strange thing the data can do — the special cases are already in there, fixed long ago by someone else, invisible to whoever compiles it today. Writing the reader fresh means meeting every one of those assumptions yourself, in the order the data chooses to break them.

So I did not set out to study data quality. I got a controlled experiment in it, by removing thirty years of accumulated defences and then pointing a brand-new consumer at a real dataset.

Roughly 9,200 lines of engine code and 2,000 of frontend, 174 engine tests, running against Freedoom — the BSD-licensed open asset set. No original source, no commercial data.

The dataset has conventions where a declarative schema should be

The game ships its content in a single container file: a header, a directory of named entries, and a defined binary layout for each kind of entry — but no declarative schema describing the semantic relationships among them. This is not unstructured data. It is structured data whose most important contracts live outside the structure. The names carry all the meaning, and every rule is unwritten:

• An entry called E1M1 is a level — but only because it is a zero-length entry immediately followed by one called THINGS. Nothing declares this. You infer it from adjacency.
• Entries beginning DS are sounds. Entries between markers named S_START and S_END are sprites.
• A sprite’s animation frame and viewing angle are encoded in the last two characters of its name, so TROO plus frame I plus rotation 1 becomes TROOI1. There is no index anywhere of which frames exist.

If you have ever consumed a partner’s CSV drop where the filename encodes the region and the date, or a bucket where _v2 in a key means the columns changed, you have met this dataset. It is enormously common — and it is not a legacy curiosity. It is what most real integrations look like before somebody writes the schema down.

Every one of those unwritten rules is a contract. And when a contract breaks, nothing throws.

Here is what that looks like from the other end — a real frame from the reimplementation, with each part of it traced back to the entry it was assembled from. There is no pixel on this screen that was not fetched by name.

Eight failures, and not one raised an error

This is the actual bug history of the project, not a constructed example:

Read the third column again. Not one of those is an exception, a stack trace, or a failed assertion. Every one is plausible wrong output. The program ran. The tests passed. The data was quietly, confidently misread.

The first row is my favourite, because it is the one that stayed fully functional while being wrong. The atlas of textures was built from the objects the level spawns — but which objects a level spawns depends on the difficulty, and the two easiest settings place two shotguns that no other setting does. So on those settings the world contained objects the atlas had never heard of. They spawned. They blocked movement. They made a sound when collected. They were simply never drawn, because the renderer skips a sprite whose texture is missing, and a missing texture is not an error.

Nobody reported it as a rendering bug. It was reported as “there is a shotgun here and I can pick it up, but I cannot see it” — which is a much better bug report than I would have written, and it names the shape exactly: the data said one thing, two consumers disagreed about it, and only one of them was visible.

That is precisely the category of bug that costs real money in real pipelines, and precisely the category that conventional unit tests written from the same assumptions are structurally bad at catching — because such a test encodes the same mental model the code does. I wrote both. They agreed with each other, and they were both wrong.

The renderer’s real job turned out to be detection

Here is the part I did not expect, and the reason I now think games belong in this conversation at all.

A renderer is a continuous, high-bandwidth assertion about your data. Point a camera at 18,000 vertices sixty times a second and a broken contract stops being an abstraction: it is a hole in a wall, a monster that is not there, a room lit wrongly. Several of those eight bugs were caught by looking at a frame — not by a test, not by a log line, not by a metric.

I want to be precise about how much luck that represents. A batch pipeline gets nothing comparable. The same eight failures inside an ETL job produce a report that is plausible, delivered on time, and wrong — and it stays wrong until somebody downstream happens to notice a number they did not expect, weeks later, if ever.

So the transferable insight is not “use a renderer”. It is this: most data systems have no comparable detector, and that absence is a design choice nobody made deliberately. If your only feedback channel is a test suite you wrote against your own assumptions, you have exactly one opinion about your data, held twice.

The discipline that worked, every time

The fix was identical in all eight cases, and it is not “write more tests”:

Enumerate what the data actually contains. Then assert that every name your code depends on is present.

Concretely, the project grew small tools whose only job is to interrogate the dataset and print what is genuinely in it:

• One lists every animation frame each character actually has. The behaviour tables were built from that output rather than from documentation or memory — and a test then demands that every frame the tables name exists in the file. Naming a frame that is not there was the invisible-monster bug. Now it fails the build.
• One lists the 69 sound entries present. A test asserts that every sound the engine can play is among them. A missing name produces silence, which is indistinguishable from the audio system being broken.
• One counts which of the roughly 140 possible level behaviours the shipped levels actually use. The answer for level one was eight. That turned an open-ended implementation backlog into a measured one, ordered by what the data demands rather than by what the specification allows.

None of that is testing the code. It is testing the code’s assumptions about the data — and in this project, that is what converted a silent gap into a build failure.

If you take one thing from this article, take that distinction. Your test suite almost certainly checks that your code does what you think. It very likely does not check that your data is what you think.

What this is not

Here is where I disappoint anyone expecting a scale story.

The numbers: 27.5 MB of content loaded once (9.8 MB compressed), a 4 MB texture atlas uploaded once per level, 0.84 MiB of vertex data per frame, a simulation stepping 35 times a second. A whole level fits in a phone’s cache. Nothing streams. There is no ingest rate, no partitioning, no backpressure worth a diagram.

Calling this a high-throughput data problem would be selling something the measurements do not support, and anyone who checks would find out in about a minute. It is a contract problem, at small volume, where the consequences happen to be visible.

Those are genuinely different problems, and conflating them is how architecture diagrams become fiction. Most organisations I have worked with have far more contract problems than throughput problems — and spend far more attention on the second.

What transfers

Strip the game out and the shape is entirely ordinary:

1. A dataset with conventions instead of a schema. Names that carry meaning, rules that were never written down, exceptions known only to whoever produced it.
2. A consumer that encodes assumptions about those conventions — necessarily, because there is nothing else to encode them against.
3. Failures that are silent, because a misread convention produces plausible output rather than an error.
4. A validation step that makes the assumptions explicit and checkable, by enumerating the data and asserting against what is actually there.

Step 4 is cheap. In this project it is a handful of small programs and a few dozen assertions — a rounding error against 9,200 lines. It is also the only reason I now trust the result.

The uncomfortable question it leaves me with, and the reason I wrote any of this down: how many of my pipelines have a detector at all, and how many merely have tests that agree with the code?

Where this came from

Two reasons, and the first is the honest one: rebuilding a game engine is enormously good fun, and I would have done it regardless of whether anything useful came out.

The second is that I needed a world to point at. I gave two talks this year arguing that a simulated world makes an unusually good laboratory for engineering autonomous systems — Building a Doom-Like World to Explore Agentic Systems at NDC Toronto, and Simulated Worlds for Agent Engineering: Planning, Policy, and Evaluation at AgentCon Toronto. The argument in both is that a game engine gives you strict control loops, complex state transitions and real-time feedback in a setting where failures are cheap and visible — and that the key invariant worth holding is that every action be observable, attributable, and reproducible through world state.

This article is deliberately about the layer underneath that claim, and I have kept it there on purpose.

Because reproducibility is not something you bolt onto a simulation. It is a property of whether the thing reading your data got it right — and a world that silently misreads its own content is not a controlled environment, however deterministic the loop above it looks. A level that spawns thirty-one objects the designer never placed is not a repeatable experiment. Neither is one whose second run draws with the first run’s textures.

So: the fun came first, the laboratory came second, and the data problem turned out to sit under both. That ordering was not planned, and it is the part I would most want someone else to take away.

What the world is for next

The engine now runs in a browser at AgentArena.ca, and that is the point at which it stops being a rebuild and starts being an instrument: one world, opened in a tab, running the same simulation on every machine that opens it.

What I intend to measure in it is agentic behaviour and implementation — how an agent perceives a frame, what it decides from that, what it actually does, and whether any of it survives being run a second time. Perception, policy and action are three separate claims, and a world with strict control loops and cheap, visible failures lets each of them be checked against recorded state rather than against a transcript of the agent describing itself.

The engine is simply the part that had to be right first, and that is the whole reason this article is about data rather than about agents. An agent evaluated inside a world that misreads its own content tells you about the world’s bugs, not the agent’s competence. Every assertion in the section above exists so that when the agent work is measured, the world is not the variable.

The one-line version

A game is a data problem wearing a graphics costume. Rebuilding one from scratch stripped away thirty years of accumulated defences and showed me eight ways a dataset can be misread with nothing reporting a problem — and that the fix is not more tests of the code, but assertions about the data, generated by enumerating what is actually in it.

The engine is original work built against Freedoom (BSD-licensed). DOOM is a trademark of id Software LLC; this project is not affiliated with, endorsed by, or sponsored by id Software or ZeniMax, and contains none of their code or data.

#DataEngineering #DataQuality #DataContracts #SoftwareArchitecture #Observability #Testing

© 2026 Alexander Chernov. All rights reserved. First published on LinkedIn, which remains the canonical version; this page is a reprint by the author.

Should You Argue With an LLM?

By Alexander Chernov. First published on LinkedIn, 2026-08-22. Read the original.

Push back on a language model and it will usually change its answer. Say “are you sure?” and it apologises, reconsiders, and hands you something different. People read this as either humility or spinelessness depending on their mood, and lately they read it as personality — the model has an attitude, the model is eager to please, the model caved.

It is none of those things in the way it sounds. It is a measured, reproducible property of these systems, and there is now enough research on it to say something practical: when arguing helps, when it quietly makes the answer worse, and why the model cannot fix this by thinking harder.

The plain measurement

The cleanest experiment is also one of the oldest. In the FlipFlop study [1], ten models were given seven classification tasks, allowed to answer, and then challenged with a single line — a version of “are you sure?” Nothing was added. No evidence, no correction, not even a hint about which direction to move. Just pressure.

The models changed their answers about 46 percent of the time. Average accuracy fell 17 points between the first answer and the last.

That is the effect in its purest form. Content-free pressure moves the answer, and it moves it downhill. Fine-tuning on synthetic data reduced the deterioration by about 60 percent and did not remove it.

Two different things are happening

A Stanford group later separated the two ways an answer can move [2]. Working across mathematics and medical questions, they found sycophantic behaviour in 58 percent of cases. But most of it was progressive — the model started wrong and the user’s pushback moved it to the right answer, in about 43 percent of cases. Regressive sycophancy, where a correct answer was abandoned for a wrong one, happened in about 15 percent.

So arguing is not simply harmful. It is a trade, and in that evaluation the odds ran roughly three to one in your favour. What the numbers do not say is what decides which side you land on. Model, version, task and the state of the conversation all move it. The one variable on your side of the exchange is why you are pushing back — whether you know something the model does not, or the answer merely made you uncomfortable. The study does not separate those two cases. It is still the distinction worth carrying into your own use of it.

Two more details from that work are worth carrying. Sycophantic behaviour persisted in about 78 percent of cases, so once the answer moves it tends to stay moved, and everything downstream in the conversation is now built on the moved answer. And disagreement stated up front, before the model answers, produced more sycophancy than disagreement raised after — 62 percent against 57. Telling the model what you think before you ask is the strongest single way to get told what you think.

Where the character comes from

The word people reach for is character, and it turns out that word is closer to the mechanism than it has any right to be.

In mid-2025, researchers at Anthropic, UT Austin and Berkeley showed that traits like sycophancy, hallucination and outright malice correspond to specific directions in the model’s internal activity — persona vectors [3]. Give the method a plain-English description of a trait and it finds the direction. Two of their findings matter for anyone typing at one of these systems.

The first is timing. The trait direction is measurable before the response is generated. Read plainly, that means the model is not deciding to flatter you partway through a sentence. Whatever it is going to sound like is already present in its state when it starts. It arrived at the answer already flattering.

The second is contamination. Character moves for reasons that have nothing to do with character. Training a model on mistaken mathematics answers made it measurably more sycophantic, more prone to hallucination, and more willing to say hostile things. A trait can be acquired from data that never mentioned the trait.

There is a smaller finding in the same family that reads like a joke and is not. Work presented at CHI this year found that grammatical person changes the sycophancy rate [4]. Ask about a claim in the third person rather than the first, and the model flatters less. Not because it is reasoning about who is asking, but because the phrasing sits somewhere else in the space, and the character comes with the coordinates.

So the attitude is real in the only sense that matters when you are working. It is a state, it is measurable, it moves, and it sits upstream of the answer. It is not a self. It is a dial that your wording is turning whether you meant to touch it or not.

Why it cannot check itself

The obvious repair is to ask the model to review its own work. Across several evaluations it has not held up. Intrinsic self-correction — reconsidering with no new information from outside — has often left performance unchanged or made it worse, across arithmetic, question answering, code generation and planning [5], [6]. Multi-agent debate, compared against the same number of samples spent on plain self-consistency, has not beaten it in those comparisons. Results vary with the model, the task and the sampling budget, but the direction is consistent enough to design around.

Self-correction does work when there is external feedback. A compiler. A failing test. A retrieved document. A person who actually knows.

That is the whole thing in one line. The answer and the check come out of the same process, so anything that pushes on the process pushes on both. There is no separate faculty in there that stands apart and audits.

So, should you argue

Yes, but with a rule: argue with evidence, never with tone.

“Are you sure?” is close to the worst input available. It carries all of the pressure and none of the information. Replace it with the reason you doubt the answer — the number that looks wrong, the source that says otherwise, the case it did not handle. Then the model has something to move toward instead of merely something to move away from. That is what raises the chance a revision is progressive rather than regressive.

State your position after the answer, not before. Ask about the claim rather than about your version of the claim. And when you cannot supply a reason and you push anyway, be honest about what is happening — that is not auditing, it is negotiating, and you will win.

Why this is about to stop being a matter of taste

In November 2025, the FDA’s Digital Health Advisory Committee met on generative AI mental health devices [7]. When the committee listed the novel risks that this class of product introduces, it named three: bias, hallucination, and sycophancy.

That is a regulator putting a conversational habit on a risk register. It is the right instinct. In a system where a person’s pushback moves the output and the moved output persists, agreeableness is not a personality quirk. It is a failure mode with a measurable rate.

Which points at the design conclusion. If the correctness of your system depends on a model holding its ground under pressure, you have built on the one property these models reliably do not have. Put the check outside the conversation. A test that runs, a source that is retrieved, a rule that is evaluated before the action commits — something with no opinion about whether you are happy. The model is a good generator and a poor witness to itself, and everything useful follows from designing around that rather than arguing with it.

References

1. Laban et al., “Are You Sure? Challenging LLMs Leads to Performance Drops in The FlipFlop Experiment”, arXiv:2311.08596.
2. Fanous et al., “SycEval: Evaluating LLM Sycophancy”, Stanford, 2025, arXiv:2502.08177.
3. Chen et al., “Persona Vectors: Monitoring and Controlling Character Traits in Language Models”, arXiv:2507.21509, and the accompanying Anthropic research note.
4. “Interaction Context Often Increases Sycophancy in LLMs”, CHI 2026, arXiv:2509.12517.
5. Huang et al., “Large Language Models Cannot Self-Correct Reasoning Yet”, ICLR 2024.
6. Kamoi et al., “When Can LLMs Actually Correct Their Own Mistakes?”, TACL, 2024.
7. FDA Digital Health Advisory Committee meeting of 6 November 2025.

#AI #LLM #AISafety #Sycophancy #AIEvaluation #AIAlignment #HumanAIInteraction #AIReliability #AIGovernance #LLMOps #MachineLearning

© 2026 Alexander Chernov. All rights reserved. First published on LinkedIn, which remains the canonical version; this page is a reprint by the author.

Every Stage Reported Success. The Data Was Wrong.

By Alexander Chernov. First published on LinkedIn, 2026-08-16. Read the original.

The failure mode I fear most in a data pipeline is not the one that pages you at 3am. It is the one where every stage returns cleanly, every row count looks sane, every dashboard renders — and the numbers are wrong in a way that is perfectly plausible.

I spent a few days recovering five years of running data off a discontinued GPS watch. It was meant to be a weekend errand. What it turned into was an unusually clean study of that failure mode, because the pipeline lied to me four separate times, and not once did anything throw.

Every one of those four was caught by the same kind of thing: a check that measured the data a different way than the pipeline produced it. None was caught by an error code, a schema, a type check, or a test that the pipeline wrote about itself.

That is the whole article. The watch is just the vehicle.

The setup, briefly

Epson sold GPS running watches, then shut the sync service down on 31 March 2025. The watch still works. Its data is still on it. The supported way to get that data off no longer exists.

Over USB the watch is not a mass-storage device — it is a vendor command protocol tunnelled over HID reports, with no published specification. Over Bluetooth LE it speaks a custom GATT protocol with its own framing and flow control, at about 180 bytes per second. I recovered both from the Android app, and the BLE path is the one that works end to end: 23 activities, roughly 2.1 MB, downloaded at the speed of a 1994 modem.

Then the interesting part. The records are in Epson’s own format — not FIT, not GPX, not anything documented.

The obvious move is to reverse the format. I did something lazier and much better: I extracted Epson’s own decoder — a native ARM64 library from inside their Android app — and ran it on my Linux laptop under a CPU emulator. No phone, no Android, no JVM. Their decoder, my machine.

That decision is worth stating plainly as a technique, because it generalises:

When a format is undocumented but a decoder for it exists, running the decoder is often cheaper and always more correct than reimplementing it. And more importantly: it gives you an oracle — a source of ground truth to check everything else against.

That oracle is what caught bug number four. Hold that thought.

Failure 1: the header that was four bytes longer than I thought

The download worked. Twenty-three records came off the watch, no errors, no timeouts, no short reads. Every file arrived.

Every file was also corrupt.

The response to each chunk request has a header, and I read that header as 10 bytes. It is 14 — status, class, element, index, then a 32-bit offset and a 32-bit length. I had accounted for the offset and not the length. So four bytes of length field stayed inside the payload, once per chunk, every 900 bytes, all the way through every file.

What made it visible was not an error. It was arithmetic: the files were 83,588 bytes against a declared size of 83,584. Four bytes too long, in a 2 MB download, on a transfer that reported complete success.

Once you look with that hint, the corruption is obvious — 84 03 00 00 repeating at a fixed stride, which is just 900 in little-endian, written into the data every 904 bytes. Without the size comparison it is indistinguishable from binary noise in a format you do not yet understand.

The lesson is cheap and general: if your source declares a size, compare it. Not because you expect a mismatch, but because a length check is nearly free and it is one of the very few things that can catch a transport bug from the outside. Byte counts are a checksum you already have.

Failure 2: the one where every latitude was zero

With the container assembled correctly, the decoder ran. It returned success. It returned 3,325 samples, 6 laps, 1,146 GPS points. Elapsed time counted up correctly. Direction, speed and cumulative distance were all present and all sensible.

Every latitude was 0. Every longitude was -2147483648 — INT_MIN, the format’s marker for “no fix”.

That is a completely coherent story. It says: this watch recorded an activity but never acquired GPS. An indoor run. A treadmill session. A watch that could not see the sky. I ran all 23 records and got the same answer 23 times, which reads as this person always runs indoors — unusual, but not impossible, and the data was internally consistent with it.

The cause was mine. To run a native library under emulation you must supply the C library functions it calls, and I had stubbed the maths functions — sqrt, sin, cos, atan2 — to return 0. Placeholders I wrote early, when I only wanted to see whether the thing would execute at all, and then forgot.

Geodesy is nothing but those four functions. Everything else in the decoder is integer bookkeeping and came out perfect. The stubs zeroed exactly one thing: position.

Wiring in real maths produced 43.749812, -79.206021. Scarborough, Ontario. Nine and a half kilometres.

This is the failure I would put on a poster:

A wrong dependency produced a complete, self-consistent, plausible dataset with one channel silently zeroed. No error. No warning. No missing rows. Just a story that happened to be false.

Note what would not have caught it. Not a schema — the field was present and correctly typed. Not a null check — the values were not null, they were 0 and INT_MIN. Not row counts, not a smoke test, not “did the job succeed”. The only thing that catches this is knowing what the data is supposed to mean and noticing that the answer, while coherent, is not credible.

Failure 3: the channel that was never there

I then wrote in my own documentation, as a statement of fact, that this watch has no optical heart-rate sensor.

I had evidence. The decoder returned no heart rate on any of the 23 activities. The output structure had no heart-rate array in it. The activities rendered fine without it. The absence was consistent, total, and explainable — plenty of older running watches use a chest strap and have no wrist sensor.

The watch has an optical sensor. The owner told me so.

The bug: the decoder’s output has around twenty container fields, and every single one of them is a {count, pointer} pair — a length and an array. So I read them all that way. One is not. The distance container has four fields, not two:

+0x00  uint32  number of distance samples
+0x08  pointer to distance samples
+0x10  uint32  number of heart-rate samples     <- never read
+0x18  pointer to heart-rate samples            <- never read

I read the first pair and stopped. The distance samples came back completely correct, because they are the first pair. Nothing was malformed. Nothing was missing from the part I looked at. The heart rate was sitting in memory, fully decoded, at an offset I never visited.

Reading it gave 23 activities with heart rate on every one.

This is the worst of the four, and it is worth being precise about why:

Absent data is invisible to every check that operates on the data you have. A validation suite examines rows that exist. It cannot flag a column you never selected. There is no anomaly to detect, no distribution to look wrong, no null to count — the pipeline is internally perfect and simply smaller than reality.

The generalisation is uncomfortable: your pipeline cannot tell you what it is not reading. Only something outside it can — a spec, an independent extract, a row count from the source system, or a human who knows the equipment. In this case it was a human, and I had already written my mistaken inference into two READMEs and a commit message as though it were a hardware fact.

Which brings up the meta-lesson. I did not just miss the data. I explained the absence, plausibly, and promoted my explanation to documentation. An unexplained gap invites investigation. A well-explained gap closes the ticket. Be most suspicious of the missing things you have a good story for.

Failure 4: the label I invented

Once heart rate was in, I generated the activity files. Each one needs an activity type, and this is a running watch, so I set them all to running.

The most recent activity was 16 km in 45:56. That is 2:52 per kilometre — about 21 km/h, which is roughly world-record marathon pace, sustained for 16 km, by a hobbyist.

Now, the honest possibility here is that the distance was wrong. So I checked it against something that had not been through the same code path: I took the GPS track — a list of latitude/longitude fixes — and computed its length with the haversine formula. Two independent measurements of the same quantity, one from the device’s accelerometer-and-GPS fusion, one from raw geometry.

They agreed to 0.7%. On another activity, 1.4%.

So the distance is right, the pace is right, and it is simply not a run. It is a bike ride. The record format carries no activity type at all — I checked, and the metadata is byte-identical across all 23 records with nothing in the header behaving like an enum.

running was not data. It was my assumption, written into the output in a field that looks exactly like data to everyone downstream.

The fix was to label them unknown, which is less useful and more true. And it is the case that most of them are runs. But a store where 22 rows are right and one is a confident lie is worse than one that admits it does not know, because nothing downstream can distinguish them.

Any field your pipeline fills in rather than reads is a claim you are making on the source’s behalf. If it is not in the source, it does not get to look like it is.

While I was at it, the same cross-check settled a units question. Cadence: was the raw value steps per minute, or steps per minute per foot (i.e. double)? The laps carry their own step counts, so — 6,799 steps across the 2,822 samples that report a cadence is 144.6 steps per minute, against the channel’s own average of 144. Not halved. Two independent paths to one number, agreeing to 0.4%.

The fifth one, in different code

I then ported the parsers to JavaScript for a browser tool, and wrote a test that generates a FIT, a TCX and a GPX file, reads all three back, and checks them against each other.

GPX came back with no heart rate. FIT and TCX were fine.

GPX stores heart rate in a namespaced extension element, <gpxtpx:hr>. My code matched elements by local name, using the DOM’s localName property, which in a conforming XML parser is prefix-free. It is not universally so. Where it was not, gpxtpx:hr stopped matching hr — and only the namespaced fields vanished. Latitude, longitude, elevation and time all parsed perfectly, because none of them are namespaced.

One format, one class of field, silently absent. Exactly the same shape as failure 3, in a completely different language and codebase, two days later.

A single-format test would have passed. The cross-format comparison caught it in one run, because it asked a question no single parser can answer about itself: do independent readings of the same activity agree?

What actually catches this class of bug

Four failures, four different mechanisms — a transport off-by-four, a stubbed dependency, a struct misread, an invented label. What caught them has more in common than what caused them:

Every one of them is a measurement taken by a different route than the one that produced the value. That is the only property they share, and I think it is the whole trick.

The practical version:

1. Compare declared to actual. Sizes, row counts, checksums. Free, and catches transport bugs from outside.
2. Compute at least one important number two ways. Distance from geometry as well as from the device. Cadence from steps as well as from the cadence channel. Agreement to a few percent is worth more than any amount of internal consistency.
3. Treat a clean absence as a finding, not a fact. Especially when you have a good explanation for it.
4. Keep filled-in fields distinguishable from read fields. unknown beats a plausible guess.
5. If a decoder for your format exists, run it. It is an oracle, and oracles are how you check the thing you wrote.

None of this is exotic. All of it is the kind of check that gets cut because the pipeline already works.

The bonus

The viewer came out of this as a small standalone thing, and it is now public:

https://doytsujin.github.io/ok-prosense-web/

Drop a FIT, TCX or GPX file and it renders the route, pace, heart rate, elevation and splits. It is a static page — your file is parsed by JavaScript in your own tab, and there is no upload, no backend, no account and no build step. Map tiles default to off, because fetching a basemap sends tile coordinates to a third party, and that would quietly undercut the one promise the page makes. It reads any watch’s export, not just the Epson one.

None of my own activities are in that repository, which left it with nothing to demonstrate on, so the README now points at public test files instead: Garmin’s own FIT fixtures, real dumps off an Edge 500 and a fēnix, TCX from a tagged set of sport activities, and a spread of GPX. Two of them are listed because they are broken in useful ways. One GPX is written as eight separate <trk> elements, which reads as an empty file to anything that takes only the first track. Another has every timestamp set to the 1901 sentinel, and the page reports that as a route with no duration rather than as a confident 0:00 over 2.7 km — the same class of lie as the zeroed latitudes above, caught the same way.

Its own test page is the part I would reuse. It generates a FIT, a TCX, a GPX and a JSON file in the browser, reads all four back, and asserts they agree — 33 assertions, no fixtures on disk, and exactly the cross-check that caught the namespace bug above. No single-format test can perform it.

The watch-specific half — the Bluetooth protocol and the emulated decoder — stays in a CLI, because decoding those records requires Epson’s own library and that is not mine to redistribute.

The part that stays with me

I got the hard things right. The Bluetooth protocol, the packet framing, the flow control, running a foreign CPU architecture’s binary under emulation and calling into it with the correct ABI — all of that worked, and none of it is where I lost time.

I lost time on four bugs that all had the same signature: the pipeline said it was fine, and it was not, and the output was plausible enough that I believed it and started writing it down as fact.

The tooling we have is very good at telling us that a job finished. It is almost silent on whether the result is true. That gap does not close with better error handling. It closes with the discipline of measuring the same thing twice, by different means, and caring when the two answers disagree.

Written up from a real recovery job: 23 activities, 214.6 km, 2021 to 2026, off a watch whose cloud service shut down last year. The viewer is linked above; the files to try it on are listed in its README.

© 2026 Alexander Chernov. All rights reserved. First published on LinkedIn, which remains the canonical version; this page is a reprint by the author.

Governing a Quantum Pipeline, Part 2

By Alexander Chernov. First published on LinkedIn, 2026-06-06. Read the original.

This is the second of two parts, and it builds directly on the first. In Part 1 I took a published quantum-genomics method — QuASeR’s de novo assembly — ran it on PennyLane as pure quantum optimisation, and wrapped it in a small control plane: a dataset descriptor governing the data, an agent lifecycle governing the work, and a gateway governing the quantum resource. Nothing in that pipeline was trained. Here I add the part I deliberately held back: the AI. The same stack that lets you solve a circuit lets you train one, so I bolt a trained quantum model onto the very same pipeline — and govern it the very same way.

If you have not read Part 1, the one-paragraph version is this: a real bioinformatics step (assembling a DNA sequence from short overlapping reads) was cast as a TSP → QUBO → cost-Hamiltonian → QAOA chain and run on nine qubits in PennyLane, reconstructing the contig GCGCA and matching the brute-force optimum. Around it sat a descriptor (schema, provenance, qubit/iteration budgets, checked before execution), a five-state agent (reason → bind → execute → publish), and a control-plane gateway (admission control, backend abstraction, a semantic circuit cache, and a structured event log). That is the pipeline this article extends.

Differentiable quantum programming: the AI side of the stack

The central idea in PennyLane is simple to state and consequential in practice: a quantum circuit can be differentiated.

You describe a circuit as a QNode, and you can then take gradients through it with the same automatic-differentiation engines that train neural networks — PyTorch, TensorFlow, JAX, or NumPy/autograd. A parameterised quantum circuit becomes, mathematically, just another differentiable block you can drop into a larger model and optimise end to end. That is what people mean when they say PennyLane lets you “train quantum circuits like neural networks.”

The mechanism underneath is worth a sentence, because it is what makes the claim more than an analogy. You cannot backpropagate through quantum hardware the way you can through a tensor in memory — a real device only gives you measurements. PennyLane gets the gradient anyway through the parameter-shift rule: to find how the output changes with a given gate parameter, it runs the same circuit twice with that parameter nudged by a fixed amount and takes the difference. The result is an exact analytic gradient that can be obtained on real hardware, not a finite-difference approximation — which is precisely why the same training loop works whether the QNode runs on a simulator or a quantum device. The autodiff engine stitches those per-gate gradients together, and gradient descent does the rest.

This is the complement to Part 1. QAOA solves a fixed problem: it tunes a circuit’s parameters to minimise one cost Hamiltonian, and when it is done you read off an answer. A trained model is different in kind — it learns from data, so that it generalises to inputs it has never seen. Both are variational circuits optimised by gradient descent; only one of them is machine learning. Part 1 exercised the first. Part 2 adds the second, on the same hardware-agnostic stack and behind the same governance.

Adding a trained stage to the pipeline

The natural place for AI in this pipeline is downstream of assembly. Part 1 reconstructs a contig; it does not interpret it. So I added a second stage that does: a variational quantum classifier (VQC) that labels the reconstructed sequence by nucleotide composition — GC-rich vs AT-rich, a real and simple genomic property.

The classifier is genuinely trained, and the construction is worth seeing in detail because every piece has a neural-network counterpart. Each sequence is reduced to a four-number feature vector — the frequencies of A, T, G, and C — scaled into angles and angle-encoded onto a four-qubit circuit (PennyLane’s AngleEmbedding). On top of that sit three layers of an entangling template (BasicEntanglerLayers), which is where the trainable weights live — the quantum analogue of a small stack of dense layers. The circuit’s output is the expectation of a single Pauli-Z measurement on one qubit, a number in [−1, 1], to which a trainable bias is added; the sign of the result is the predicted class. Training minimises a plain mean-squared-error loss between that output and ±1 targets, using the Adam optimiser — the identical recipe you would reach for with a classical model, except the forward pass runs a quantum circuit and the gradients come from the parameter-shift rule above.

What it trains on matters as much as how. The model learns from a balanced, labelled set of short DNA sequences — equal numbers of GC-rich and AT-rich examples, with the ambiguous borderline around a 50% GC fraction deliberately excluded so the label is never in doubt — split into a training portion and a held-out test portion it never sees during optimisation. Reporting accuracy on that held-out split, rather than on the data it trained on, is the difference between a model that learned and one that merely memorised. Applied to the contig GCGCA that Part 1 assembled, the trained circuit returns GC-rich, correctly and with a comfortable margin.

It is a small, honest task. The point is not that a quantum classifier beats a one-line if statement on GC content — it obviously does not. The point is that a trained quantum model is now a first-class stage in the pipeline, sitting immediately downstream of a quantum optimiser, both written against the same PennyLane front-end. The data-to-AI-to-quantum arc stops being a slogan here: it is one workflow in which the same gradient machinery runs a solver and a learner back to back.

Governed exactly like the optimisation stage

The part I care most about is that adding AI did not mean adding a second, ungoverned escape hatch. The trained stage runs under the same control plane Part 1 built, with the differences you would expect made explicit rather than hidden:

• Its own descriptor. The classification stage carries its own dataset descriptor — output schema (GC-rich / AT-rich), provenance that names the trained-VQC method and links back to the assembly stage as its source, and resource policies. Where the assembly descriptor bounded a qubit budget and an iteration budget, this one bounds a qubit budget and a training-step budget — a learned model’s distinctive cost is the training, so that is what the policy governs, checked before the stage runs.
• The same agent lifecycle. The classifier stage is driven by the same reason → bind → execute → publish agent. It evaluates its training and qubit budgets in the reasoning phase, before it trains anything, and it publishes the label with full provenance. The reasoning-before-execution barrier that made the optimisation stage safe to automate makes the AI stage safe to automate for the same reason.
• The same gateway. The classifier’s circuit is admitted through the same control-plane gateway as the QAOA circuit — the same admission control against the tenant’s allowed backends, the same rate limits, the same semantic cache (classify the identical contig twice and the second is served from cache), and the same structured event record. The AI does not get a private door to the quantum resource.
• The same contract discipline. As with assembly’s “reconstruction matches the optimum,” the AI stage publishes against a contract — the predicted label is checked against ground truth, and the trained model has to clear a held-out-accuracy floor before its output is accepted. A model that trained badly fails its contract rather than silently shipping a guess.

There is a quieter payoff in that first bullet. Because the classification descriptor names the assembly stage as its source, the two stages are not just adjacent — they are linked, and the link is recorded. The provenance now reads as a chain: raw reads → the QAOA-assembled contig → the trained-VQC label, each step naming the one before it and the method that produced it. That is what lets you answer, after the fact, not only “what did the AI predict” but “what was it predicting about, and where did that come from” — the question that governance exists to keep answerable when a learned model sits in the middle of a pipeline.

Run the full two-stage pipeline and the behaviour is exactly the union of the two stories: the assembly is admitted and executes, the trained classifier labels the contig and passes its contract, an identical resubmission is served from the cache, and a request for an unapproved backend is denied before it reaches the device. Two stages — one that solves a circuit, one that trains one — under a single, auditable control plane.

The observation

Strip away the branding and PennyLane is a useful lens on where quantum computing is heading: hardware-agnostic, compiler-backed, autodifferentiation-native, and openly reproducible. It treats a quantum program as differentiable software that lives inside the ordinary machine-learning toolchain — and it exposes many different machines through one interface.

That is what “quantum is becoming a software discipline” looks like when you write it in actual code. And taken across both parts, it is where the data-to-AI-to-quantum progression stops being a slogan. Part 1 showed that a quantum optimisation stage can be run like a governed production workflow. Part 2 showed that a trained quantum model slots into that same workflow under the same governance — that AI and quantum are not separate stories bolted together, but two stages of one optimisation loop, governed alike. The interesting frontier is not only making quantum circuits trainable; it is making the whole hybrid pipeline — optimiser and learner together — something you can operate, audit, and trust.

A self-contained, runnable companion accompanies the series: the QuASeR QAOA chain, the trained VQC classifier, and the shared descriptor runtime and gateway. The default run exercises both stages of this Part 2 pipeline (python3 governed_stage.py); the Part 1 pipeline alone is python3 governed_stage.py --stage1-only.

References: PennyLane (GitHub): https://github.com/PennyLaneAI/pennylane PennyLane: https://pennylane.ai/ QuASeR — Sarkar, Al-Ars, Bertels, PLoS ONE 16(4):e0249850 (2021): https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0249850

#QuantumComputing #PennyLane #QuantumMachineLearning #DifferentiableProgramming #QuantumSoftware #ControlPlane #AgenticDatasets #OpenSource

© 2026 Alexander Chernov. All rights reserved. First published on LinkedIn, which remains the canonical version; this page is a reprint by the author.

Governing a Quantum Pipeline, Part 1

By Alexander Chernov. First published on LinkedIn, 2026-06-03. Read the original.

This is the first of two parts. Here I take a real quantum-genomics method, run it on an open quantum stack, and wrap it in the governance a production workflow would actually need — as pure quantum optimisation, with nothing trained. Part 2 adds the AI: the same stack that lets you solve a circuit also lets you train one, and I bolt a trained quantum model onto this same pipeline, governed the same way.

In an earlier piece I argued that quantum computing is becoming as much a software and platform discipline as a hardware one, and that data, AI, and quantum increasingly build on one another rather than standing alone. I named several frameworks only in passing — among them PennyLane, the open-source stack from Xanadu. PennyLane is the one that deserves a closer look, and the cleanest way to look at it is not to admire the SDK but to run something real through it and ask what it takes to operate that responsibly.

So that is what these two articles do. This is an observation about a stack and a way of working, not an endorsement of a product.

PennyLane: one front-end, many backends

What makes PennyLane interesting as infrastructure is that its programming surface sits on top of a real engineering stack, not a single notebook.

• PennyLane is the programming surface — circuits expressed as ordinary Python functions (QNodes), reusable circuit templates, and quantum-chemistry tooling for building molecular Hamiltonians.
• Catalyst is a just-in-time compiler for hybrid quantum-classical programs, built on the MLIR compiler infrastructure. The presence of a real compiler layer is itself the tell: quantum programming has become, in part, a compiler-engineering problem.
• Lightning is a family of high-performance state-vector and tensor-network simulators written in C++, with GPU execution — the unglamorous, essential plumbing that makes iteration fast.
• Plugins connect the same front-end to many hardware backends — IBM through Qiskit, IonQ, Alpine Quantum Technologies’ ion traps, Amazon Braket, and photonic devices. You write once and retarget.

That last point is the pattern I keep returning to: one front-end, many backends, behind a single uniform interface. It is the same abstraction discipline that makes any heterogeneous platform governable — here applied to quantum hardware that is otherwise wildly different underneath. (There is a second thing this stack lets you do — differentiate and train a circuit, not just run it. That is the AI side, and I am deliberately holding it back for Part 2. Part 1 is about the solve side and how to govern it.)

A published quantum-genomics method

To make this concrete, I took an actual pipeline from the literature and ran it on the PennyLane stack. QuASeR (Sarkar, Al-Ars and Bertels, PLoS ONE, 2021) performs de novo assembly — reconstructing a DNA sequence from many short, overlapping fragments (reads) without a reference genome to align them to. It is one of the first steps in a sequencing pipeline, and a hard one. QuASeR’s move is to treat assembly as an optimisation problem and hand that problem to a quantum computer.

One thing to flag up front: this stage is quantum optimisation, not machine learning — nothing here is trained. The chain is short, but each link is a genuine translation:

• the reads and their pairwise overlaps become a travelling-salesman problem (TSP): find the order that strings the fragments together with the most overlap, i.e. the shortest sequence that contains them all;
• the TSP becomes a QUBO — a quadratic unconstrained binary optimisation, the 0/1-variable form that quantum optimisers accept (here a variable for “read i sits in slot p”);
• the QUBO becomes a cost Hamiltonian — an energy function whose lowest-energy state encodes the best ordering;
• and that Hamiltonian is minimised by QAOA, the Quantum Approximate Optimisation Algorithm, a variational circuit that alternates the cost and a mixing operation and is tuned to concentrate probability on low-energy (good) solutions. PennyLane implements QAOA directly.

I built that chain in PennyLane — qml.qaoa, a qml.device, a @qml.qnode — on a deliberately small instance: three short reads that tile the sequence GCGCA.

It works, within the limits of the experiment. On this small instance the QAOA circuit — nine qubits — returns a distribution whose most probable feasible ordering (a valid permutation of the reads) is the optimal assembly, reconstructing GCGCA and matching the brute-force optimum. This is a demonstration on a tiny instance, verified rather than asserted: QuASeR is explicit that QAOA’s accuracy is limited at depth, and nothing here claims a quantum speedup. The point is narrower and, I think, more useful — a real bioinformatics task runs, end to end, on the open quantum stack.

Running the method is only half the story

The other half is everything around it — and that is where data, agents, and orchestration come in. (I use agent here in the orchestration sense — a controller that runs a stage, evaluates its policies, and records what happened — not in the machine-learning sense; there is no model or LLM in this loop.) I wrapped the assembly stage in two layers that together behave like a small control plane for a hybrid quantum-classical job.

The data, described

A dataset descriptor is a compact record that travels with the data and states what it is and how it may be used: its schema (the shape of the data), its provenance (where it came from and by what process), and the policies that constrain it. For this stage the descriptor carries resource policies — a qubit budget and an iteration budget — that are checked before the job runs. The shift is small to state and large in consequence: the data stops being an inert input and starts carrying its own rules. A dataset that knows its own schema, lineage, and limits can be validated, audited, and refused automatically, instead of relying on whoever happens to be running the pipeline to remember the constraints. That is what I mean by an agentic dataset — data that participates in its own governance rather than waiting to be acted upon.

The agent, and why a lifecycle

An agent runs the stage, but not as one opaque function call. It moves through an explicit lifecycle — reason → bind → execute → publish — and the order is the point. In the reasoning phase it looks up the descriptor and evaluates the policies before anything irreversible happens; only if they pass does it bind its inputs and execute; afterwards it publishes the result together with its provenance. This reasoning-before-execution barrier is what makes autonomy safe to grant: a step that can act on its own first needs a defined moment at which it decides whether it should, and a record of that decision. Every transition is emitted as structured telemetry, so the run is observable in detail after the fact — not merely flagged as succeeded or failed.

The orchestration, and the control plane

The second layer governs the quantum resource itself. In front of the backend sits a gateway — a control plane in the sense the term carries in distributed systems: it does not do the science, it decides what is allowed to run, where, and at what cost, and it keeps the system’s invariants. Concretely it does four things. It performs admission control: a job is admitted only when the requesting tenant’s policy allows the chosen backend, so an unapproved device is refused before a circuit ever reaches it. It provides a backend abstraction: the same job description can target a local simulator or a real machine, which is exactly what lets a workflow move from simulation to hardware without rewriting the science. It keeps cost bounded: rate limits, plus a semantic cache keyed on the circuit itself, so an identical circuit is served from cache instead of re-running — fewer redundant, billable shots on real hardware. And it records every decision as a structured event, so the orchestration is auditable as a whole, not just per job.

Those are the questions a control plane for quantum has to answer that a bare SDK does not: which workloads are admitted, how cost stays bounded across providers, and how the whole orchestration remains correct and auditable. The gateway here is a deliberately minimal version — admission, backend abstraction, caching, and event logging, not reconciliation or multi-provider scheduling — but it is the same shape, and it is enough to show why the shape matters.

The behaviour is what you would want from such a system. An approved request on an allowed backend is admitted, the QAOA assembly runs, and a contract confirms the reconstruction matches the optimum. Submit the identical circuit again and the gateway serves it from cache — no second execution. Point a request at a backend the tenant is not cleared to use and the gateway denies admission before the job reaches the device; the stage faults and the denial is in the record. Nothing reaches the quantum resource that the policy did not allow.

Where Part 1 lands

That is the whole idea in miniature, with a real method underneath it: a published quantum-genomics algorithm, implemented on PennyLane, running as a policy-checked, observable, reproducible stage — the descriptor governing the data, the agent governing the work, and the control plane governing the quantum resource. No AI yet, and that is deliberate. The pipeline earns its governance on the optimisation side first.

But QAOA is only one of the two things this stack lets you do to a circuit. The same gradient machinery that solves a circuit here can train one — and a trained quantum model is genuinely machine learning, not optimisation. In Part 2 I add exactly that: a trained variational quantum classifier as a second stage, downstream of this assembly, governed by the very same descriptor runtime, agent lifecycle, and control plane. That is where AI meets the pipeline.

A self-contained, runnable companion accompanies the series: the QuASeR chain on real PennyLane, plus the descriptor runtime and the gateway. To reproduce the Part 1 pipeline — assembly and governance, no AI — run the demo in its stage-one mode (python3 governed_stage.py --stage1-only).

References: PennyLane (GitHub): https://github.com/PennyLaneAI/pennylane PennyLane: https://pennylane.ai/ QuASeR — Sarkar, Al-Ars, Bertels, PLoS ONE 16(4):e0249850 (2021): https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0249850

#QuantumComputing #PennyLane #QuantumOptimization #QAOA #ControlPlane #AgenticDatasets #QuantumSoftware #ResearchInfrastructure #OpenSource

© 2026 Alexander Chernov. All rights reserved. First published on LinkedIn, which remains the canonical version; this page is a reprint by the author.

Physics Reproducible by Agentic Construction

By Alexander Chernov. First published on LinkedIn, 2026-05-31. Read the original.

I once contributed to a paper — a piece of classic strong-field physics — on how an electron escapes an atom under a two-colour laser field. The physics still holds. What I want to revisit is not the result but the process — because the way I would build that same research today, as a governed and reproducible workflow, is completely different from the pen-and-paper way it was actually done. So this is an old result used as a worked example of an agentic research workflow: first what it says, then how I would build it now.

Years ago, with two colleagues, I helped derive an analytical expression for the ionization of an atom in a bichromatic laser field — a fundamental frequency together with its second harmonic. The result generalized the well-known Keldysh formula to the two-colour case, and it showed something pleasing: adding the second harmonic, with the right phase and intensity, can sharply enhance ionization. Coherent control, on paper.

I am not here to re-litigate the physics. I am here because the process is the interesting part now. So let me do two things — explain what the result actually says, and then rebuild it the way I now build research infrastructure: as an agentic, reproducible workflow rather than a derivation you have to re-read in order to trust.

What the result is about

Start with a bound electron sitting in the potential well of an atom. Shine an intense laser on it and something counter-intuitive becomes possible: the electron can tunnel out. The oscillating electric field, added to the atom’s own potential, bends the wall that holds the electron in — far enough, for long enough, that there is a finite probability of escape. And because the field oscillates in time, that wall is not static. It breathes, rising and falling each optical cycle.

Whether “tunnelling” is even the right picture is governed by a single dimensionless number: the Keldysh adiabaticity parameter γ. When γ is small, the field changes slowly compared with the time the electron needs to cross the barrier, and the process really does look like tunnelling through a quasi-static wall. When γ is large, the wall flickers too fast for that picture, and ionization is better described as the simultaneous absorption of many photons. The strength of the Keldysh framework is that one expression bridges both regimes.

Now make the field two-colour: a strong fundamental at frequency ω plus its second harmonic at 2ω, with a relative strength α = E₂/E₁ and a controllable relative phase. The second harmonic breaks the symmetry of the field — it makes one half-cycle push harder than the other — and that asymmetry is a knob you can turn. Using the imaginary-time method (a standard technique for tunnelling problems, in which the electron’s path under the barrier is followed in imaginary time), we derived the escape probability in the form

D = exp(−(q²/ω)·F(γ, α)),

where F is a “tunnelling exponent” that carries all of the physics and the under-barrier time is fixed by a transcendental equation. The smaller F is, the larger the ionization probability D.

Any result of this kind has to respect its limits, and checking them is half the work:

• with the second harmonic switched off (α → 0), F must collapse back to the original monochromatic Keldysh result — and it does;
• in the strong-field, slowly varying limit (small γ) it must reproduce the familiar static-field tunnelling exponent — and it does;
• and in between, the second harmonic interferes constructively and lowers F, which raises the ionization probability. That enhancement — coherent control of ionization by a second colour — was the point of the paper.

Where this physics is put to work today

It would be easy to read all of this as a museum piece. It is not. The specific move at the heart of the result — shaping a laser field out of two colours to control how an atom ionizes — turns out to be one of the central tools of modern ultrafast and strong-field physics. A few of the places it shows up:

• Attosecond science and high-harmonic generation. Tunnelling ionization is the first step of high-harmonic generation: an electron is freed, driven back by the oscillating field, and recombines, emitting a burst of extreme-ultraviolet light. Two-colour ω + 2ω fields — exactly the bichromatic configuration here — are a standard way to break the field’s sub-cycle symmetry, steer that recollision, and carve out the isolated attosecond pulses that let us film electron motion in real time. The relative phase between the two colours is the control knob.
• Terahertz generation from two-colour plasmas. Focus a fundamental and its second harmonic together into a gas, and the asymmetric field drives a net electron drift current as it ionizes — and that current radiates intense, broadband terahertz light. The ω + 2ω asymmetry that enhances ionization in this result is the very same asymmetry that makes two-colour air plasma one of the most widely used table-top THz sources, with uses from spectroscopy to security imaging.
• Imaging molecules with their own electrons. Because tunnelling ionization is so sensitive to the field and to the orbital the electron leaves from, it has become a probe: laser-induced electron diffraction and high-harmonic spectroscopy use the ionized-then-returning electron to read out molecular structure and watch bonds rearrange on femtosecond-to-attosecond timescales. The Keldysh framework that this result extends is still the language those measurements are interpreted in.
• Coherent control, more broadly. The underlying principle — that interference between two driving pathways (here ω and 2ω) controls the outcome of a quantum process — is the same idea behind coherent control of photoionization, and even of chemical reactions: tune a phase, and you bias where the electron goes or which channel opens. Directional control of photoelectron emission with ω + 2ω fields is now a routine diagnostic.
• Lightwave electronics. At the frontier, the same ability to steer electrons with a tailored optical field, faster than a single cycle of light, is what “petahertz” or lightwave electronics is built on — driving and reading ultrafast currents in gases and solids with shaped fields. Two-colour control of ionization is one of the simplest members of that family.

None of this depends on the particular formula we derived; the point is that the physics it describes — adiabaticity, tunnelling, and two-colour coherent control of ionization — is alive and applied. A result that looked, at the time, like a clean analytical curiosity sits upstream of attosecond metrology, table-top THz sources, and molecular movies.

How that research happened — and what was left implicit

At the time, this was pen-and-paper work. The derivation was the artifact. The limiting cases were our tests, but we ran them in our heads and wrote them up in prose. Reproducibility lived in the printed equations: if you wanted to check the result, you re-did the algebra. Provenance was the reference list. There was no shared code, no dataset, no environment to re-run — and that was completely normal. A good analytical paper was exactly this.

It worked. But notice what was implicit. The assumptions — a single active electron, a particular relative phase, a short-range approximation that ignores the Coulomb tail — together with the checks and the chain from equation to number, all lived in the prose and in our heads. Nothing carried them but the paper itself. If a number was wrong, or an assumption mattered more than we thought, nothing would tell you. You had to already know.

The same result, today, as an agentic workflow

So I rebuilt the result — not to improve the physics, but to run it the way I now build research infrastructure. The organizing idea is the agentic dataset: instead of a result being an inert number in a table, it is data that carries its own context and can take part in its own validation.

Concretely, the reproduced result carries four things:

• a descriptor — what it is, the schema of its outputs, and the assumptions that were once implicit, now written down explicitly;
• a contract — the guarantees the result must satisfy to count as valid at all;
• provenance — a content hash of the data together with a fingerprint of the exact code that produced it, so any stored number can be traced back to the computation that made it;
• triggers — conditions that fire automatically: change the parameter grid and the contract re-runs, because stale data should never pass silently.

Those four are not independent labels; they interlock, and that is what makes the dataset active rather than passive. The descriptor is what makes the once-implicit assumptions addressable — “single active electron”, “zero relative phase”, “short-range potential” become named fields you can point at, not caveats buried in a paragraph. The contract is what turns the descriptor’s promises into something a machine can check, and it travels with the data, so the guarantees are not a separate test file someone has to remember to run. Provenance is what lets a downstream result notice it has gone stale: a content hash that no longer matches is a signal, not just a receipt. And triggers are what close the loop, so the dataset can ask to be re-validated when its inputs move, refuse to be used outside the range its contract actually covers, and tell whatever depends on it that it now needs to relearn. The shift is small to state and large in consequence — from a number you have to remember the caveats for, to data that carries and enforces its own caveats. A dataset that knows its own schema, limits, and lineage can be validated, audited, and refused automatically, instead of relying on whoever happens to be running the pipeline to remember the rules.

The limiting cases we once checked in our heads become executable contracts — and they double as the test suite:

• α → 0 reproduces the Keldysh function f(γ) to machine precision (10⁻¹⁶);
• the small-γ static-field limit converges to its known value;
• the second harmonic lowers F, so the ionization probability D rises — in the demo, by a factor of about 10⁷ as α goes from 0 to 1;
• and every stored number is re-derivable from the pinned code, exactly.

The computation itself runs as a staged pipe — ground → compute → validate → record. Ground fixes the inputs and assumptions; compute evaluates the physics; validate runs the contracts; record emits a provenance artifact alongside the output. The stages are explicit and ordered for the same reason a build pipeline is: each one has a defined job, and the result is “published” only once it has passed validation. None of that is specific to physics — it is the same descriptor / contract / provenance / pipe pattern I use for sprawling enterprise data, here applied to a classic, decades-old equation.

The payoff is concrete. The figure below is produced by the workflow — it is not pasted in. Re-run the pipe and the figure, the numbers, and the provenance artifact regenerate together, each stamped with the code fingerprint that made them.

The demo itself is small — the physics, an agentic-dataset wrapper, the contracts, and the pipe, with the contracts doubling as CI tests. The interesting part isn’t the lines of code. It’s that the result now travels with the evidence that it is correct.

Could an LLM agent help here?

Worth being precise: there is no language model anywhere in this demo. The “agent” that drives the pipe is a small deterministic controller — it evaluates policies and records what happened, and nothing about it is intelligent. But it is fair to ask where an LLM-based agent would genuinely earn its place in a workflow like this, because the honest answer is “in a few specific spots, and always behind the contracts.”

The natural ones are all drafting and triage — the bookkeeping, not the judgment:

• Writing the descriptor from the prose. The assumptions that were implicit in the original paper are exactly the sort of thing a language model is good at surfacing: read the derivation, propose the schema and the list of assumptions, and hand a physicist a draft descriptor to correct rather than a blank form to fill in.
• Proposing the contracts. The limiting cases are stated, in words, in the paper — “it must reduce to Keldysh as α → 0.” An LLM can turn those sentences into candidate executable checks, which a human approves before they ever become gates.
• Explaining a failure. When a contract trips, an agent that can read the telemetry and the provenance can draft a diagnosis — “the small-γ check drifted after the grid changed; suspect the under-barrier solver tolerance” — and shorten the loop from red to understood.
• A natural-language way in. “What assumptions does this result rest on? Is it still valid at γ = 2?” is answerable straight from the descriptor and the contracts; an LLM is a reasonable interface to that — as long as it answers from the governed artifacts, not from its own memory.

The thread through all of those is a single rule: the LLM proposes; the control plane disposes. Its descriptor draft is still validated, its suggested contract still has to be approved and then actually pass, its failure diagnosis is a hypothesis the checks confirm or refute. And that is the quietly useful part — the agentic-dataset machinery is exactly what you need to keep a probabilistic assistant honest. A language model is fluent and confident and sometimes wrong; contracts, provenance, and triggers are precisely the apparatus that lets you accept its speed without trusting its say-so. You let it draft, and you let the executable checks decide. That division of labour — LLM for the language, contracts for the truth — is the same one the next section draws for automation in general.

Where agents help — and where they don’t

I want to be careful here, because this is where the hype usually goes wrong. Automation and AI agents are very good at the bookkeeping: carrying the derivation, running the checks, hashing the provenance, regenerating the figure when an input changes. They are not the ones who decide that a single-active-electron model is the right idealization, or that the constructive-interference regime is the physically interesting one, or that the result is worth believing. That judgment is still the physicist’s.

This is the same point I keep coming back to: the tool does not remove the need to understand the problem. It removes the excuse for the problem to be irreproducible.

The systems takeaway

A classic result, reproducible only by re-reading the algebra. The same result today — described, contract-checked, provenance-stamped, and re-runnable by anyone. The physics did not change. What changed is that the process became an infrastructure question, exactly the shift from data, to AI, to governed scientific workflows that I keep writing about.

That is what I mean by agentic research workflows: datasets that carry their own contracts and provenance, pipelines that validate themselves, and results that arrive with their evidence attached. The derivation was finished long ago. The reproducibility is what I would build today.

#AgenticAI #ResearchInfrastructure #ReproducibleResearch #ComputationalPhysics #DataEngineering #ScientificComputing #Provenance #QuantumControl

© 2026 Alexander Chernov. All rights reserved. First published on LinkedIn, which remains the canonical version; this page is a reprint by the author.

From Data to AI to Quantum-Enabled Biology

By Alexander Chernov. First published on LinkedIn, 2026-05-30. Read the original.

I’ve been looking at the work coming out of The University of Osaka’s Center for Quantum Information and Quantum Biology — QIQB.

What caught my attention was less the quantum hardware itself than the broader research architecture around it. QIQB brings together several areas that are often discussed separately: quantum computing, quantum information devices, quantum communication and security, quantum measurement and sensing, and quantum biology.

That combination matters because the future of quantum research goes beyond building better devices. It is equally about building the surrounding infrastructure that makes those devices useful: cloud access, software stacks, experiment control, data workflows, security, reproducibility, and interdisciplinary collaboration.

It also points to a wider arc I want to trace here — the movement from data, to AI, to quantum-enabled biology — and why each step in that arc builds on the infrastructure beneath it rather than standing alone.

QIQB: https://qiqb.osaka-u.ac.jp/en/

Osaka University Quantum Computing Cloud: https://www.qiqb-cloud.jp/

OQTOPUS: https://oqtopus-team.github.io/

Why QIQB Stands Out

Public sources point to several notable elements: QIQB’s six research areas, the OQTOPUS open-source quantum cloud stack, a QIQB quantum-cloud portal, and the July 2025 launch of a domestically developed superconducting quantum computer at QIQB.

The striking part is that it shows quantum computing as more than a laboratory capability — it is becoming a platform capability.

That distinction is important.

A laboratory capability may demonstrate that a machine exists and can run experiments. A platform capability asks a bigger question:

How can researchers, students, engineers, and application developers access, use, reproduce, govern, and integrate this capability into real scientific workflows?

That is where the systems dimension begins.

From Quantum Hardware to Quantum Infrastructure

The most visible part of quantum computing is usually the hardware: superconducting qubits, trapped ions, photonics, cryogenic systems, microwave control, optical control, error mitigation, and experimental stability.

But hardware alone is not the full story.

To make quantum systems useful across disciplines, we also need infrastructure around the hardware:

• cloud execution environments;
• open-source software stacks;
• scheduling and workload management;
• experiment control;
• compilers and transpilers;
• hybrid quantum-classical workflows;
• error mitigation;
• access control;
• provenance and auditability;
• reproducibility;
• integration with classical high-performance computing and data platforms.

This is why the OQTOPUS direction is notable. An open-source stack for cloud-based quantum computers suggests that quantum computing is becoming as much a software, operations, and platform engineering discipline as a physics and hardware challenge.

That is a major shift.

OQTOPUS also does not stand alone. It joins a growing ecosystem of frameworks for actually writing and running quantum programs: IBM’s Qiskit, Google’s Cirq, and Xanadu’s PennyLane for hybrid quantum-classical and quantum machine learning, alongside managed cloud services such as Amazon Braket and Microsoft Azure Quantum that expose several hardware backends behind a single interface. The fact that a researcher today reaches first for a software development kit, a transpiler, and a cloud endpoint — rather than a cryogenic lab — is itself a clear sign of how far quantum computing has moved toward a software and platform discipline.

Qiskit (IBM): https://www.ibm.com/quantum/qiskit

Amazon Braket (AWS): https://aws.amazon.com/braket/

How Data, AI, and Quantum Build on One Another

One reason this topic interests me is that modern biology is increasingly becoming a data-intensive discipline.

Genomics, proteomics, imaging, electronic health records, lab automation, molecular simulation, and biomedical literature all produce massive and heterogeneous data streams. The challenge is no longer collecting data. It is making that data usable, trustworthy, reproducible, and actionable.

This is where AI has already changed the landscape.

Machine learning and foundation models are being used to search literature, classify biomedical signals, analyze images, predict molecular properties, support drug discovery, and assist with biological interpretation.

But AI systems also introduce new infrastructure requirements:

• data provenance;
• model governance;
• traceability;
• validation;
• monitoring;
• reproducibility;
• careful control over automated decisions.

Quantum computing and quantum sensing add another layer to this evolution.

In biology, quantum-related capabilities may become relevant through quantum chemistry, molecular simulation, nanoscale sensing, optimization, and hybrid quantum-classical workflows. These capabilities do not replace classical data platforms or AI systems. Instead, they may become specialized components inside larger scientific workflows.

What makes this powerful is that the layers compound. Each one builds on the one before it. Usable, well-described data is what makes AI dependable. Governed AI is what makes a quantum resource worth pointing at a real biological question. And a reproducible workflow is what turns any single result into something other researchers can build on. The progression from data to AI to quantum is therefore additive, not a hand-off: data plus AI plus quantum adds up to capability that no single layer delivers on its own.

That is why the connection from data to AI to quantum computing matters.

The future research environment may not be a single model, a single database, or a single quantum computer. It may be an orchestrated scientific platform where biological datasets, AI models, simulation tools, quantum resources, lab instruments, and human researchers interact through governed workflows.

In that kind of environment, infrastructure becomes central.

We need systems that can answer practical questions:

• What data was used?
• Which model or simulation produced this result?
• Which assumptions were applied?
• Was a quantum resource used, and under what conditions?
• Can the result be reproduced?
• Was the workflow approved, monitored, and governed?
• Can another researcher understand the full chain from biological question to computational result?

This is where I see a direct connection to intelligent control planes, dataset descriptors, provenance, and policy-governed scientific computation.

Quantum biology may be an emerging field, but the data and infrastructure questions it raises are already familiar: how to make complex scientific workflows reliable, explainable, reusable, and trustworthy.

Why Quantum Biology Is a Careful but Important Direction

Quantum biology should be discussed carefully.

It should not be treated as a shortcut to claim that quantum computers are already transforming all biological research. That would be too sweeping.

But the research direction is important.

Biological systems involve molecular interactions, energy transfer, electronic structure, sensing, noise, and dynamics at very small scales. Some of these questions naturally touch quantum chemistry, quantum measurement, and nanoscale sensing.

It is tempting to ask:

Can quantum computers solve biology on their own?

A more realistic and productive question is:

How can quantum methods, quantum-inspired methods, quantum sensing, and high-quality computational infrastructure improve the way we study biological systems?

That question connects quantum biology with computational biology, AI, data engineering, and scientific infrastructure.

The Canadian Context

This topic is also relevant in Canada.

The University of Waterloo’s Institute for Quantum Computing is one of Canada’s major quantum research hubs, bringing together science, mathematics, and engineering around quantum information science and technology.

Institute for Quantum Computing, University of Waterloo: https://uwaterloo.ca/institute-for-quantum-computing/

The University of Toronto’s Centre for Quantum Information and Quantum Control is a second important Canadian reference point. It frames quantum research across physics, chemistry, mathematics, computer science, electrical engineering, and related disciplines.

Centre for Quantum Information and Quantum Control, University of Toronto: https://cqiqc.physics.utoronto.ca/

This matters because quantum research increasingly depends on interdisciplinary capacity. It spans far more than physics: computer science, electrical engineering, materials science, chemistry, mathematics, security, cloud infrastructure, and eventually domain-specific scientific applications.

Canada’s expanding quantum ecosystem is well positioned for this kind of interdisciplinary work.

IEEE Quantum Week / QCE 2026 in Toronto

A timely connection is IEEE Quantum Week 2026, also known as the IEEE International Conference on Quantum Computing and Engineering — QCE 2026.

The 2026 event is scheduled for September 13–18, 2026, at the Metro Toronto Convention Centre in Toronto, Ontario, Canada.

IEEE Quantum Week / QCE 2026: https://qce.quantum.ieee.org/2026/

What makes QCE compelling is that it sits exactly at the intersection implied by the name: quantum computing and engineering.

That engineering dimension matters.

As quantum systems mature, the field needs more than theoretical breakthroughs and hardware demonstrations. It needs practical engineering around:

• cloud access;
• software tooling;
• compilers and transpilers;
• hybrid quantum-classical workflows;
• benchmarking;
• education and workforce development;
• security;
• reliability;
• reproducibility;
• application integration.

In other words, quantum computing is becoming a systems discipline.

The Systems Question

For me, the most interesting part isn’t the quantum hardware itself.

It is the systems question around it:

How do we make advanced scientific infrastructure accessible, observable, governable, and usable by researchers across disciplines?

That question applies to quantum computing, but also to computational biology, biomedical data platforms, AI-driven research workflows, and large-scale scientific data infrastructure.

It connects directly to several themes I care about:

• reproducible computational biology workflows;
• policy-governed data pipelines;
• provenance and auditability;
• intelligent control planes for scientific computation;
• agentic research workflows;
• data products that can trigger validation, relearning, and analysis;
• infrastructure that is explainable enough to be trusted.

This is why the QIQB example sticks with me.

It is about more than a quantum computer. It is about the underlying pattern of scientific infrastructure becoming cloud-accessible, software-defined, interdisciplinary, and workflow-driven.

Closing Thought

Quantum computing is often discussed as a future technology.

But the infrastructure questions are already present now.

Who can access these systems? How are experiments submitted and reproduced? How are results tracked? How are workflows governed? How do we connect quantum systems with classical computing, biomedical data, chemistry, sensing, and AI?

These are not secondary questions. They are part of making quantum technology practically useful.

That is why I will be watching QIQB, Canada’s quantum ecosystem, and IEEE Quantum Week / QCE 2026 with interest.

Handled as one governed, reproducible system, data, AI, and quantum infrastructure stop being separate stories. They build on one another — a compounding advance for biology.

#QuantumComputing #QuantumBiology #ScientificComputing #ComputationalBiology #ResearchInfrastructure #AI #DataEngineering #IEEEQuantumWeek #QCE2026 #CanadaQuantum #OpenSource

© 2026 Alexander Chernov. All rights reserved. First published on LinkedIn, which remains the canonical version; this page is a reprint by the author.

AI Amplifies People Who Know How to Think

By Alexander Chernov. First published on LinkedIn, 2026-05-10. Read the original.

AI is becoming the new calculator.

That may sound like a simple comparison, but I think it is a useful way to understand where AI fits into the future of work, education, engineering, research, and decision-making.

A calculator does not remove the need to understand mathematics.

It helps you calculate faster. It reduces repetitive manual effort. It allows you to check results, explore alternatives, and focus on higher-level reasoning instead of spending all your time on arithmetic.

But a calculator is only useful when the person using it understands what they are trying to calculate.

If you do not understand mathematics, a calculator can still produce an answer — but you may not know whether that answer makes sense. You may not recognize when the wrong formula was used, when the input was incorrect, or when the result is technically valid but practically meaningless.

AI is similar.

AI can help us write, code, analyze, summarize, design, research, and reason faster. It can accelerate knowledge work in the same way calculators accelerated numerical work.

But AI does not remove the need for domain knowledge.

Without subject-matter understanding, we may not know whether an AI-generated answer is correct, incomplete, biased, outdated, overconfident, or simply irrelevant.

The real skill is not only “using AI.”

The real skill is combining AI fluency with domain expertise.

That means knowing how to ask precise questions. Knowing how to provide context. Knowing how to evaluate the answer. Knowing when to challenge the output. Knowing when something looks plausible but is wrong. Knowing how to connect the result to a real problem, real constraints, and real consequences.

In engineering, this matters.

AI may generate code, but engineers still need to understand architecture, reliability, security, maintainability, and failure modes.

In data science, AI may summarize patterns, but practitioners still need to understand data quality, causality, bias, assumptions, and interpretation.

In research, AI may help synthesize literature, but researchers still need to understand methods, evidence, uncertainty, and what is actually novel.

In business, AI may generate recommendations, but leaders still need to understand trade-offs, risks, strategy, and accountability.

about:blank#blocked

The calculator did not eliminate mathematics.

It changed what became valuable.

Manual calculation became less important. Mathematical understanding, modeling, interpretation, and problem formulation became more important.

AI will likely do the same for knowledge work.

It will reduce the value of some repetitive tasks. But it will increase the value of clear thinking, strong judgment, domain expertise, and the ability to validate results.

AI will not remove the need to think.

It will raise the value of people who know how to think clearly, ask precise questions, and verify what they receive.

The future is not simply AI replacing expertise.

The future is expertise amplified by AI.

#AI #ArtificialIntelligence #Engineering #DataScience #FutureOfWork #MachineLearning #DigitalTransformation #KnowledgeWork

© 2026 Alexander Chernov. All rights reserved. First published on LinkedIn, which remains the canonical version; this page is a reprint by the author.

From Protocols to Control Planes: What MCP Teaches Us About Building Agentic Systems at Scale

By Alexander Chernov. First published on LinkedIn, 2026-03-29. Read the original.

I was recently selected to speak at Optimized AI Conference 2026 and the MCP Dev Summit North America 2026.

Due to logistical constraints, I was unable to attend. However, the material prepared for these sessions reflects a direction I believe is becoming structurally important for modern AI systems.

This article summarizes that perspective.

The Initial Assumption: MCP as a Lightweight Protocol

Model Context Protocol (MCP) is often introduced as a simple abstraction:

a stateless interface for tool invocation between agents and external capabilities.

At small scale, this mental model holds.

Agents call tools. Tools respond. The system behaves predictably.

What Actually Happens in Production

Under real-world conditions, this abstraction begins to break.

Once deployed in environments with:

• multiple concurrent agents
• dynamic capability evolution
• Kubernetes-based execution
• continuous deployment and scaling

MCP stops behaving like a protocol.

It starts behaving like a distributed system.

And more specifically:

a control plane.

The Structural Shift

This transition is not accidental—it is structural.

The moment you introduce:

• session continuity across unstable infrastructure
• replay and idempotency requirements
• shared capability catalogs
• multi-tenant policy constraints

you are no longer designing a protocol.

You are designing a system that must coordinate state, enforce rules, and maintain consistency across independent actors.

In other words:

you are building a control plane.

The Failure Modes (Observed and Reproducible)

When this shift is not acknowledged, systems tend to fail in predictable ways:

1. Session Instability Under Infrastructure Churn

Pods restart. Routes change. Agents lose context.

Without a control-plane layer, session continuity becomes fragile.

2. Replay and Idempotency Gaps

Agent interactions are not inherently safe to replay.

This leads to duplicated actions, inconsistent outcomes, and non-deterministic workflows.

3. Capability Catalog Drift

Tools evolve. Interfaces change.

Without strong control-plane governance, different agents operate against inconsistent views of the system.

4. Backpressure and Overload

As concurrency increases, systems lack mechanisms to regulate flow.

The result is cascading failure rather than graceful degradation.

5. Multi-Tenant Policy Ambiguity

In shared environments, enforcing who can do what—and under which conditions—becomes a first-class concern.

This cannot be reliably handled at the protocol level.

A More Accurate Mental Model

A more robust way to think about MCP in production is:

• MCP provides minimal primitives

• A control plane layer manages:

  • routing
  • policy enforcement
  • consistency
  • observability
  • lifecycle management

This mirrors patterns seen in:

• service meshes
• event-driven systems
• actor models
• distributed data platforms

Toward Agentic Control Planes

This perspective has been shaping my ongoing work around:

• agentic datasets (datasets as active participants in execution)
• descriptor-driven systems (declarative control surfaces)
• policy-aware execution models
• large-scale multi-agent environments (10⁶+ agents)

The key idea is simple:

decisions, policies, and capabilities should be treated as first-class control-plane artifacts—not implicit side effects of agent execution.

Why This Matters

As AI systems move from isolated workflows to persistent, multi-agent environments, the cost of ignoring control-plane design increases dramatically.

Without it:

• reproducibility degrades
• governance becomes reactive
• system behavior becomes difficult to reason about

With it:

• systems become observable
• policies become enforceable
• behavior becomes auditable and controllable

Closing

Although I wasn’t able to present this work in person, I’m continuing to develop these ideas in both research and production contexts.

I believe this transition—from protocol thinking to control-plane thinking—will define the next generation of agentic infrastructure.

I’m looking forward to contributing these directions in future venues.

© 2026 Alexander Chernov. All rights reserved. First published on LinkedIn, which remains the canonical version; this page is a reprint by the author.