Bolt AI into your loop and you might just have become a medical device manufacturer

The thing that makes open source AID worth having is that the code is sitting there in front of you. If you want to know why the algorithm cut your basal at four o’clock this morning you can go and read the function that decided it, and if you disagree with what it does you can argue with the people who wrote it, or fork it and change it yourself. Nobody is asked to take a manufacturer’s word for anything, which is not a small thing when the software under discussion is the software dosing you.

That same openness is also an open door, and across both of the oref-based platforms people have been walking through it for years. Most of what gets forked is small and private and never goes anywhere much. Every so often, something becomes the thing that everybody ends up running.

What has changed over the last year or so is what people are forking these systems for. Language models have become cheap enough and fluent enough that AI hubs are starting to appear in the ecosystem, bolted on to help with tuning settings and counting carbohydrate, and the pitch behind them is an easy one to make. Your data is already sitting in a Nightscout instance and a query costs pennies, while reviewing a fortnight of pump data is exactly the sort of tedious pattern-matching that most of us do badly, and usually only after something has already gone wrong.

So it reads as an enhancement, and in some ways it genuinely is one. It also carries risks that are not visible from the outside, several of which we have written about here already. What I want to do in this piece is pull those together and work out how they shift depending on which of the three obvious ways you choose to build the thing. Then I want to say where I think the line sits between writing a helper and manufacturing a medical device, because it is not where most people building these features appear to think it is, and only one of those three shapes crosses it.

The three jobs are not the same

It is worth separating out what is actually being asked for here, because “an AI hub” bundles together work that has very little in common beyond the fact that a language model is doing it.

Counting carbohydrate from a photograph ends in a number that somebody types into their app, and that number becomes a bolus. When it is wrong the damage is bounded by whatever max bolus happens to be set to, and it gets partly cleaned up by the loop over the hours that follow. The person is also awake at a table, looking at the food in question, while all of this goes on.

Settings advice is a rather different creature. Change somebody’s ISF and you have changed every automated correction from that point onward, including the ones that fire at four in the morning while they are asleep, so there is no bounded blast radius and no moment at which anyone is watching. The feedback arrives over days rather than hours, and the parameters interact with one another in ways that make even a plausible-looking single change hard to reason about. That is much of the reason there are some fifty algorithm preferences sitting behind one of these systems, on top of the time-segmented schedules for basal, sensitivity, ratio and target, and why most of them are deliberately awkward to reach.

Mealtime management sits between the two and mostly resolves into the first one with extra steps. There is also a fourth job that nobody has asked for, which I happen to think is the only one that is unambiguously worth doing, and I will come back to that at the end.

What we’ve learned about Large Language Models in diabetes this year

I have spent most of this year measuring these things, and two results are worth carrying into any conversation about building one.

The first is that asking a vision model how much carbohydrate is on a plate does not get you a stable answer. Send the same photograph five hundred times with the randomness turned right down and a plate of paella comes back at anything between 55 grams and 484 grams, which at one unit per ten grams is the difference between a snack and an ambulance. The obvious response is to sample a few times and show the range rather than a single figure, which works right up until you meet a cheese sandwich with the carbohydrate printed on the wrapper. Three models settled on twenty eight grams for a forty gram meal, one of them five hundred and ten times at a coefficient of variation of a third of one per cent. A narrow range around a wrong number reads as confidence.

The second is that if you hand five models a week of CGM and pump data and ask for settings, what comes back is mostly their training data rather than yours. Every one of them recommended a duration of insulin action of four or five hours to people running nine or ten, and nine times out of ten a recommendation sat closer to that model’s own average across all three people than to the person it was analysing.

That study ended with two recommendations for anybody deploying this. Give the model the person’s current settings, because without them it anchors on the textbook. And check what it cites, because the glucose values these models quote back at you in support of their reasoning are wrong about a third of the time, in a format that looks exactly like somebody reading a chart properly.

In August a paid Android app appeared that packaged your Nightscout data and profile into a tidy file, handed you prompts for whichever chatbot you fancied, and checked what you brought back before rendering it as a settings review. It did both of the things above, so on paper this was the good version of the idea, and I ran its own workflow four hundred and ninety times across eleven models on my own record.

The recommendations worked: the models read the file properly, and essentially every figure they quoted was genuinely in it. What did not improve was anything they concluded from it. One model lowered my maximum insulin on board in forty nine conversations out of fifty while proposing eighteen different pairs of values for it, and two others produced another twenty two pairs between them. Four models changed nothing. Another rewrote my sensitivity factor to fourteen different values between thirty and a hundred and eighty against a current fifty six, which is the difference between doubling the strength of every correction I make and cutting it by two thirds. Running it once, you get one of those answers, stamped as checked, with nothing on screen to say it was a single card from a deck. The developer withdrew the app and refunded subscribers after I published.

Structure fixes what the model reads, and leaves entirely alone what the model concludes.

That is the finding that matters here. Tidy the input and the reading improves, which is worth having. Where a fortnight of pump data admits more than one honest reading, though, and it usually does, the conclusions still diverge, and which of them you see is settled by sampling inside a model you did not choose.

Three ways to embed AI and what each one lets you fix

So to the three shapes on offer. You copy your data into whichever chatbot you like and paste the answer back, or you bring your own API key and let the app talk to the model on your behalf, or the project runs the service and the whole thing simply works. The intuition about these runs backwards, and I think that matters more than anything else in this piece. Copy and paste feels safest because the project never touches a model, bring your own key feels reasonably safe because the user is paying and therefore somehow owns the consequences, and the hosted hub feels dangerous because the project is plainly on the hook.

Clinically, none of that moves. Every failure I have described lives inside the model and turns up identically in all three, whether that is the sampling variance, the confident misidentification, the anchoring on textbook values or the decorative citations. The paella spread does not shrink because you pasted the photograph in yourself.

What does change is how much you can do about it, and that splits into two things which do not arrive together: how much control you have over the question you ask, and whether you ever find out what came back.

Copy and paste gives you neither. You cannot sample the model five times and show a range, you cannot check the glucose values it cites against the CGM record you are already holding, you cannot bound what comes back or stop it proposing changes to the safety limits, and you cannot pin a version so the thing behaves next month as it does today. Its one virtue is friction, in that nothing is prefilled and the person retypes every number by hand, so the clipboard quietly enforces a confirmation step you would otherwise have to remember to build.

Bring your own key hands you the first and none of the second. You own the prompt, the context assembly, the parsing and the display, so you can call the model five times in parallel and show the spread, verify every glucose value it cites against your own database before it reaches the screen, and put maximum basal, maximum IOB and maximum bolus permanently beyond the model’s reach. That last one matters more than it sounds, because a clamp only protects you from outside the loop, and the moment your AI can propose a new ceiling the ceiling has stopped being one. What you cannot do is learn what any of it did. The answer goes to that person’s phone and nowhere else, you never see a response one of your users actually received, and you do not know which model they pointed it at, because the key is theirs. When a provider ships an update that quietly changes your feature, you find out from nobody. You have written the question with great care and you are blind to every answer.

The hosted hub is the only one of the three where you can find out whether the thing works. You pin a model and a version, log what was suggested against what people did about it, watch a distribution rather than a single draw, and roll back within the hour when something turns. That is not a refinement on the other two. It is the difference between shipping a feature and shipping one you can stand behind. It also carries the failure the others do not: when a clinician gives poor advice it affects their list, whereas one bad model version in a hub everybody uses reaches everybody at once, and you would learn about it through adverse events rather than monitoring unless you’d built the monitoring first.

Where the regulator stands

We have been through what makes software a medical device here before, so this is the short version. It turns on intended purpose, read from what the software is held out as doing rather than from where the arithmetic happens, and MDCG 2019-11 is explicit that the location of processing makes no difference. Rule 11 puts software informing therapeutic decisions in Class IIa, or IIb where a wrong decision could cause serious deterioration, which severe hypoglycaemia does.

None of which is in any doubt for the software under discussion. An AID system automates the delivery of insulin, the documentation says exactly that, and nobody in this community has ever pretended otherwise. By function it is a medical device and not a low-class one, and it was that long before anybody attached a language model to it. The defence has never rested on what the software is.

It rests on one thing, which is that nobody is placing it on any market. The regulation is built to govern manufacturers making devices available to other people. What sits in these repositories is source code, and source code is not a device made available for use. The person who compiles it is the one doing the manufacturing, they are doing it for themselves, and an individual building a device for their own use is not what the framework was written to reach. No binary moves, no money moves, and nobody is supplied.

This definition sorts the three shapes into two piles that do not match what they might seem.

Copy and paste and bring your own key stay on the right side of it. Shipping a prompt inside the source places nothing on any market, and the person still compiles the app themselves exactly as they did before. The user’s own key makes no difference in either direction either, since who pays for inference is a billing arrangement and the regulation has no interest in billing arrangements. These two carry the clinical risk this piece has spent most of its length on, and the second moves a fortnight of somebody’s health record to a third party, which are both good reasons for care. Neither moves you across the line.

The hosted hub moves you across it. Operate a service and something is being supplied to a population of people, and the fact that every one of them compiled their own client does not unsupply it. You would be distributing a medical device even though nobody ever downloaded an app from you, and the DIY position that protects the loop does not stretch to cover a service running alongside it. It is also the point at which your prompt stops being purely a safety question and starts being the statement of what your service is for, so an instruction that lets the model describe what the data shows while forbidding it to propose a value for any setting is worth having twice over.

Handing out binaries breaks the same test by another route. Fork one of these systems, close the source and distribute builds through TestFlight or another mechanism, and compiled software has gone to people who did not build it. An internal tester group makes that starker rather than softer, since internal testers have to be added as users on the distributor’s own developer account, leaving Apple holding a roster of exactly who was supplied. Apple’s guideline 1.4.2 holds that drug dosage calculators must come from the drug manufacturer, a hospital, university, health insurance company, pharmacy or other approved entity, or receive approval by the FDA or one of its international counterparts, because it needs to be sure the app will be supported over the long term. Nobody distributing a fork is any of those things.

Which leaves the bind, and I have not found my way out of it. The only shape that lets you discover whether your feature is safe is the hosted one, and you discover it by keeping records of what the model suggested and what people did about it. That is post-market surveillance, and it is also a standing database of glucose, insulin and settings for your whole user base, which needs a controller who is somebody in particular rather than a project. So the shape that lets you behave responsibly is the same one that makes you a manufacturer and a data controller in a single step, and the two that keep you clear of both guarantee you never find out what your users were told and whether your system is safe.

What would be worth building?

The safest use of a language model in an AID system is the one nobody has asked for, and it has been sitting in the code the whole time, which is to ask it to explain what the loop already did. The determine-basal function in oref is open and deterministic, and it generates its own reasoning strings as it goes, and “why on earth did it do that at three in the morning” is genuinely the thing people find baffling about these systems. There is no new path to insulin anywhere in that feature, every claim it makes about your data can be checked against your data automatically, and it addresses a real confusion rather than manufacturing a decision that nobody needed to make.

Carb estimation is defensible provided you are honest about it. That means ranges rather than single numbers, the identified food shown and confirmed by the person before any figure appears at all, nothing ever prefilled into a field, and the whole thing off by default, in the way these projects already default their outbound features off. The residual risk is a confident misidentification of the food itself, which I cannot engineer away and neither can you, and people deserve to be told that in a sentence rather than have it buried in a disclaimer they will never read.

Settings generation I would not ship, and not primarily on regulatory grounds, although those point the same way. I would not ship it because we now have the measurement from a product that did everything my previous study recommended, and whose users still received a single draw from a distribution they were never shown. Forty different pairs of safety limits from one record, sensitivity factors spanning a factor of six. No prompt fixes an absence of information in the input. For anyone who does not log meals there is simply nothing in the data from which a carb ratio can be derived, which the model will tell you itself if you read its reasoning carefully, before handing you a number anyway.

And so

That whole study cost me somewhere near a hundred and fifty pounds in API credit, which is a weekend and a couple of tanks of petrol rather than a research budget, and it is available whichever of the three shapes you pick. Anybody shipping one of these could do that much before release and publish what came back. It is the floor rather than the ceiling, mind, because one record on one person against the models that happened to exist that week tells you a good deal less than it feels like it does.

The openness is the whole point of this, and I would not want any of it read as an argument against it. It is what lets any of us read the algorithm and argue with it, and it is why these systems are as good as they are. It also means nobody has ever stood between an enthusiastic fork and the people who install it, which has been fine for as long as the changes were the sort of thing another developer could read. A language model is not that sort of change. You cannot read it or predict it, and after a year of measuring the things I would not assume that asking the same question twice gets you the same answer.

So build what you like. Fork it, bolt a model onto it, run the thing on yourself and find out what it does to you. That is what the licence is for, and it is nobody’s business but yours.

The shape that lets you behave responsibly is the same one that makes you a manufacturer and a data controller in a single step.

Build all of it or part of it for anybody else and you have become a manufacturer, and it makes not the slightest difference that you never charged them a penny. Forking has always meant the right to change how your own insulin gets dosed. It has never meant the right to change how somebody else’s does.