I Spent a Weekend Giving an AI Eyes. It Already Had Them.

Saturday, October 10, was slow, so I asked myself a simple question: can Jev make images?

Jev is TypeSafe's decision model. You give it some text and a list of options, it picks one, and it does that in roughly a tenth of a second for a fraction of a cent. I already use it in jevvium to choose the next tap while it explores a mobile app. Drawing seemed like a stretch. Seeing seemed impossible: the docs said text only.

I spent the weekend building it a pair of eyes anyway. Four days later, TypeSafe announced that the model had them the whole time. They were just switched off.

This is that weekend. Mostly failures, a few real wins, and one very funny HTTP header.

Step 0: is it really blind?

Before building anything I wanted proof. I made three solid squares, red, blue and green, and sent each one to Jev in four different request shapes, asking what color it was.

It said "red" every time.

The image bytes got billed as text tokens, and a top-level images field came back as a 400. Blind, confirmed. So every experiment after this followed the same rule: code turns pixels into words, Jev decides, code paints.

Round 1: drawing

The first idea was the dumbest one: a 16x16 canvas, one color decision per pixel. I asked for a red heart on a white background and got this:

................
................
................
...RRR.RR.R.....
...RRRR.R.......
...RRRRR........
...RRRRR........
...RRRRRR.......
...RRRRRR.......
...RRRRRRR......
...RRRRRRR......
...RRRRRRRRR....
...RRRRRRRRRRR..
...RRRRRRRRRRRRR
.R.RRRRRRRRRRRRR
RRRRRRRRRRRRRRRR

A red wedge. Every pixel was a reasonable local decision, and together they meant nothing.

Shapes went much better. Jev picks the next element ("a sun", "a house wall", "a roof"), then its shape, color, size and place ("on top of the house wall"), and a small layout engine turns those words into coordinates. Code then describes what got drawn ("18% of the canvas wide, covers 10% of the sun") and Jev can adjust it.

A house, a snowman, a car, a flower and a boat at sunset. Peak fridge-door art.

Many of the failures were mine, not Jev's. I never told it the canvas color, so it drew a white snowman on a white canvas. I never said "lower-left part of", so the car got one wheel. The interesting result was somewhere else: when I generated four versions and let Jev pick the best one from their descriptions, its pick beat a random version by about a point out of 10 (Claude scored them blind). Choosing between options is what Jev is built for, and it showed.

Round 2: photo editing

Next: "fix this photo so it looks natural." Code describes the photo in words (brightness, contrast, saturation, warmth, a 5x5 grid of color names), Jev picks one of 11 edits and a strength, and code applies it.

On my one test photo, a macOS wallpaper, it looked solved. Then I built a benchmark: 24 real photos, 9 kinds of damage, 3 rounds, scored by distance to the undamaged original. Leaving the photo alone scores 18.7. Every Jev mode scored worse, between 19 and 24. An oracle that knows the original gets about 11, so the edits themselves were fine. What was missing was knowing what each photo should look like.

Two lessons I already knew from my day job in testing, and had to learn again:

• Never trust one test image. Everything works on the example you tuned it on.
• Jev is only as smart as the words you give it. My labels were tuned on a vivid wallpaper, so they called most real photos "dull", and Jev saturated everything.

Round 3: borrowing someone else's eyes

So I cheated a little. Claude looks at each photo once and lists what's in it, with rough boxes. Code cuts the photo into hundreds of small regions that follow its edges, and Jev answers yes or no for each one: is this the thing? Code smooths those answers into a mask and edits only inside it.

That worked. Grapes turned green, skies turned into sunsets, and coffee cups disappeared:

"Remove the coffee cup": the original, what Jev selected (magenta goes, cyan is what the hole gets filled from), and the result.

Cool, but honestly, that's Claude seeing and Jev following. Not what I wanted to prove.

Round 4: a retina made of text

The real question was whether Jev could see with no other model at all. So I wrote a retina in plain code. It cuts the image into 12 to 36 segments of similar color and turns it into text:

• a map: a grid of letters, one letter per segment
• a legend for each letter: color and RGB, size, position, what it borders or sits inside
• a silhouette drawn with # and ., like this triangle:

.....##.....
...######...
.##########.

Then I generated 30 random scenes of circles, squares and triangles and asked Jev questions with known answers:

Counting only worked when Jev answered "what is B?" for each segment and code did the adding. Asked in one go, it was close to chance. Same lesson as jevvium: let the model make small decisions and let code do the bookkeeping.

Then I had it act on what it saw. Over 60 scenes with instructions like "remove the triangle" or "make the blue one yellow", it picked the right shape 97% of the time and the right new color 30 out of 30.

Eight instructions, before and after. Jev chose the shape, code moved the pixels.

On real photos, big regions work too. With only the retina, "make the sky look like a sunset" selects the sky exactly:

The original, what Jev selected, and the result. Photo by Paul Jarvis on Unsplash.

The wall: real objects

Small objects in real photos are where it fell apart. I asked it to find 18 things across 7 photos: a coffee cup, a phone, a notebook, a license plate, a bunch of grapes. It found 5. It kept picking the biggest, most central thing in the room, and at one point it confidently called a laptop screen "the phone".

Red: Jev's pick, a miss. Green: Jev's pick, a hit. Dashed white: the right answer.

That makes sense. Text made by code carries shape, color and layout. It doesn't carry what a cup looks like. I was basically speedrunning computer vision from before deep learning, and losing.

The plot twist

On October 14, TypeSafe shipped Jev 1.14. Among its new experimental "danger zone" features: images.

And then this line, about the exact model I had spent my weekend on:

"Surprise: the Jev you all know and love (jev-1.13.0) was multimodal all along!"

The eyes were there the whole time. You just have to send one header:

TypeSafe-Danger-Zone-Images: true

My red-square test wasn't wrong. Without that header, Jev really is blind. I tested the switch, not the model.

So yes, I spent a weekend building a worse version of a feature that already existed, behind a flag, in the model I was using. That's it. That was the whole weekend.

Was it a waste?

I don't think so. Rebuilding a bad version of vision taught me more about how vision works than reading about it would have. And most of what I learned has nothing to do with images:

• Benchmarks beat demos. The wallpaper said "done". The 24 photos said "worse than doing nothing".
• Stopping belongs to code. When the checks passed, Jev still wanted one more edit. jevvium works the same way: the model explores, and the expect checks decide pass or fail.
• "Leave it" needs its own question. As one option among eleven, it almost never wins. Asked first as a yes or no, it does.
• Ask small questions, then add up. Jev counts badly in one go and well one segment at a time.

The obvious next experiment is a rematch: the same 30 scenes and the same 18 objects, through the real eyes this time. TypeSafe itself warns that models, Jev included, do worse with images than with text, so my retina might not lose by as much as I expect. Or it will lose badly, and I'll write about that too.

Everything is open, failures included, and every number here has a script behind it.

Links

• iJev, the code behind this post: github.com/AndresCarreonDiaz/iJev
• jevvium, where I use Jev for mobile testing: github.com/AndresCarreonDiaz/jevvium
• TypeSafe's Jev 1.14 announcement: typesafe.ai/blog/jev-1-14
• How to send images to Jev: docs.typesafe.ai/go/images
• The benchmark photos come from picsum.photos (Unsplash License).