What this is
A Turkish adaptation of “Do Androids Laugh at Electric Sheep? Humor ‘Understanding’ Benchmarks from The New Yorker Caption Contest” (Hessel, Marasović, Hwang, Lee, Da, Zellers, Mankoff, Choi — ACL 2023). The original asks whether a model can match a caption to a cartoon, tell a winning caption from a losing one, and explain why a joke is funny. This adaptation keeps all three tasks and both settings, but rebuilds the content for Turkish.
Why not just translate the original?
Translating New Yorker captions would measure translation, not humor. The paper's own claim is that winning captions rely on “indirect and playful allusions to human experience and culture” — translate “Can you please pass the cow?” and the incongruity survives but the cultural ground disappears. So all 10 scenes here were written from scratch in settings that only work in Turkey: the fare passed hand-to-hand in a dolmuş, endless tea in a kıraathane, the grocer's veresiye notebook, a job interview conducted by reading coffee grounds, a mangal lit in the emergency lane, gold pinned on the couple at a wedding. Each caption needs a specific piece of cultural knowledge to be solvable — and that knowledge is recorded explicitly in the dataset, so failures can be split into “didn't know the custom” vs. “knew it but reasoned wrong”.
The three tasks — üç görev
Same structure as the paper. In the from description setting the model gets a written scene annotation (location, literal description, what is uncanny); in the from pixels setting it gets the cartoon image instead.
1 · Matching
Five captions, only one was written for this cartoon. The other four are real captions from other cartoons in the set.
Cartoon: mahalle bakkalı (corner grocer)
- A) Çayları sen ısmarlamıştın galiba.
- B) Adınız hangi ciltte, hatırlıyor musunuz? ✓
- C) Ben de sadece tahlil sonucu bekliyorum.
- D) Altınlar sayılana kadar oyun havasını uzatın.
- E) Buhar öğleden sonra kesiliyor, yarın gelin.
2 · Quality ranking
Two captions written for the same cartoon — one a finalist, one not. Which one made the final?
Cartoon: hastane bekleme salonu (hospital waiting room)
- A) Yerinizi kaybetmeyin, bir daha sıra vermiyorlar. ✓
- B) Sağlık sisteminde bekleme süreleri gerçekten çok uzun.
The loser states the joke outright; the finalist plays it deadpan straight.
3 · Explanation
Free-text: explain why the winning caption is funny. Scored by pairwise human preference against a written reference, as in the paper — not automatically.
“Fincanda bir yol görünüyor ama bizim şirkete çıkmıyor.”
Reference mentions that in Turkish coffee fortune telling a “road” means travel — and that the line fuses fortune-teller talk with a corporate rejection.
Try the matching task yourself — kendin dene
All 10 matching items, exactly as the models saw them in the from pixels setting: the cartoon, five captions, no written description. Pick one.
The 10 cartoons — karikatürler
Drawn from code (SVG → PNG) rather than licensed from a magazine, so the set is copyright-clean and reproducible with one command. Badges show whether a vision model, given the image alone and no captions, correctly identified what is odd in the scene — 6 of 10 passed.
“Arkadan bir tam geliyor.”
“One full fare coming from the back.”
What's odd: Elden ele uzatılan şey para değil, dolmuşun yarısı kadar büyük bir kaya parçası. Yolcuların hiçbiri bunu tuhaf bulmuyor, herkes rutin bir şey yapar gibi davranıyor.
In a dolmuş (shared minibus) passengers pass their fare hand-to-hand up to the driver, announcing it out loud.
“Bir tane daha içelim, sonra kalkıyoruz.”
“Let's have one more, then we're off.”
What's odd: Boş çay bardaklarından oluşan kule tavana kadar uzanıyor ve tavanı delip yukarı devam ediyor. Garson kulenin tepesine yeni bardağı koyabilmek için bir el merdivenine tırmanmış.
Tea in a kıraathane (traditional tea house) never stops; “one more and we'll go” almost never ends in going.
“Nem sorunu gündemin altıncı maddesi, sırası gelince konuşacağız.”
“The damp is item six on the agenda, we'll get to it in turn.”
What's odd: Toplantı salonu diz boyu suyla dolu. Sandalyeler, oturan insanlar ve yöneticinin dosyası suyun içinde; kimse buna aldırmıyor, herkes gündem maddelerini takip ediyor.
Turkish apartment owners' meetings run strictly by numbered agenda items.
“Adınız hangi ciltte, hatırlıyor musunuz?”
“Which volume is your name in, do you remember?”
What's odd: Veresiye defteri, kütüphanelerdeki el yazması ciltler gibi devasa; ayrı bir ahşap rahleye konmuş ve bakkal sayfayı çevirebilmek için merdivene çıkmış.
The corner grocer's veresiye notebook records customers' running debts, on trust alone.
“Fincanda bir yol görünüyor ama bizim şirkete çıkmıyor.”
“There's a road in your cup, but it doesn't lead to our company.”
What's odd: İncelenen şey adayın özgeçmişi değil, ters çevrilmiş bir kahve fincanı. Duvarda kurumsal organizasyon şeması yerine büyük bir fincan-telve diyagramı asılı.
In Turkish coffee-ground fortune telling, seeing a “road” means travel or a new direction in life.
“İkametgâh için önce kaderinize bakmam gerekecek.”
“For a residence certificate I'll need to read your fate first.”
What's odd: Muhtar sandalyede değil, yerden yükselen buhar bulutunun içinde bağdaş kurmuş oturuyor; antik bir kâhin gibi gözleri kapalı. Oda kâğıt kokulu bir tapınağa benziyor.
The muhtar is the elected neighbourhood official who stamps mundane paperwork.
“Işık yeşile döndüğünde köfteler hazır olur.”
“The meatballs will be done by the time the light turns green.”
What's odd: Emniyet şeridinde mangal kurulmuş, kilim serilmiş, semaver kaynıyor. İki araba arasına çamaşır ipi gerilmiş ve üstünde çamaşırlar asılı; kurulan düzen geçici değil, yerleşik bir hayata benziyor.
Setting up a mangal (barbecue) is the default Turkish response to any outdoor wait.
“Yeni bakkal biraz yüksekte.”
“The new grocer is a bit high up.”
What's odd: Sepetin ipi aşağıya değil yukarıya gidiyor; bulutların arasında kaybolup görünmez oluyor. Kadın ipi rahatça çekiyormuş gibi duruyor.
Residents lower a basket on a rope to the grocer below — money down, bread up.
“Yerinizi kaybetmeyin, bir daha sıra vermiyorlar.”
“Don't lose your place, they won't give you another slot.”
What's odd: Ekranda sıra numarası yerine bir yıl yazıyor: 2074. Bekleyenlerden biri artık iskelet hâlinde, ayağında hâlâ hastane terliği var; yanındaki hasta sakince gazete okuyor.
Losing your place in a hospital queue means losing the appointment itself.
“Çeyrek takanlar sağdan, gram takanlar soldan.”
“Quarter-coin givers to the right, gram-gold givers to the left.”
What's odd: Takılan altınların ağırlığından sahnenin ortası çökmüş; gelin altın liralardan örülmüş bir zırh giyiyor. Salonun kapısında zırhlı para nakil aracı ve iki güvenlik görevlisi bekliyor.
At weddings guests pin gold on the couple; the amount is a public status signal.
Results — ilk ölçüm
Models served over an OpenAI-compatible endpoint, temperature=0,
three runs per task. Each cell shows the three run scores out of 10.
| Model | Matching (chance 2) | Ranking (chance 5) | Matching — location only |
|---|---|---|---|
| gpt-oss-120b | 10, 10, 10 | 9, 8, 9 | 8, 7, 5 |
| qwen3.6-35b | 10, 10, 10 | 9, 9, 9 | 7, 7, 7 |
| gemma-4-31B-it | 10, 10, 10 | 10, 10, 10 | 9, 8, 8 |
Same model, same items, the only difference being whether the scene arrives as text or as a picture:
| gemma-4-31B-it | Matching | Ranking |
|---|---|---|
| from description (text) | 10, 10, 10 | 10, 10, 10 |
| from pixels (image) | 8, 8, 8 | 7, 7, 7 |
What we learned — bulgular
- The matching task is saturated — that's a flaw in the benchmark, not a model result. All three models scored 10/10 on all three runs. An ablation shows why: strip the description and the uncanny annotation, leave only the location word (“dolmuş”, “kıraathane”), and accuracy still sits at 5–9 out of 10 against a chance level of 2. Most of the signal is topical word-matching between caption and setting, not humor. The cause is that all 10 cartoons are in different places, so distractors are trivially eliminable. Fix: draw distractors from cartoons sharing the same setting — which needs 4–5 scenes per location, i.e. a 40–50 scene pool.
- One item is probably mislabelled. On
sr01two of three models preferred the non-finalist over the gold finalist, in every run. That distractor refers directly to the odd object in the scene, and models seem to favour captions that name the anomaly. Until a real vote settles it, the item is untrustworthy. - Explanation is where models actually separate — and all of them make errors. Two models invented a pun that does not exist in the Turkish (claiming “tam” means both “full fare” and “a whole rock”), and attributed the line to the driver instead of a passenger. One named the custom correctly but got when the phrase is used wrong. This mirrors the error pattern in the original paper: on the right track, wrong in the details.
- Vision un-saturates the benchmark, but partly for the wrong reason. The from-pixels setting drops 10→8 and 10→7, reproducing the paper's finding that pixels are much harder than descriptions. But a separate legibility test — captions withheld, model asked only what the scene is and what is odd — shows the anomaly readable in 6/10 images and the setting in only 4/10. The task errors land on exactly the unreadable scenes. So the drop mixes “humor is hard” with “this drawing is ambiguous”, and cannot be cleanly attributed.
- Stick figures cannot carry a culture. The scenes that fail are the culturally specific ones — the dolmuş interior reads as a bus with a cracked window, the kıraathane as an underground bunker, the wedding hall as a circus stage. Adding signage would fix it instantly but would turn setting recognition into OCR. A real visual benchmark needs licensed or commissioned artwork.
temperature=0is not deterministic here, and not equally so per model. One model returned identical results across three runs; another swung 8/7/5 on the same ablation. With 10 items a single item is 10 points, so single-run comparisons are meaningless.
Before this could become a real benchmark
- Grow to 200–700 scenes, with 4–5 per location so distractors can be setting-matched.
- Get caption quality labels from an actual public vote, not author judgement.
- Commission or license real cartoon art; gate every image through the legibility test above.
- Collect a human reference score from Turkish native speakers — without a ceiling, model numbers cannot be interpreted. The original paper's human matching score was 94%.
- Report ≥3 runs per model with mean and spread; split cross-validation by cartoon.