IA idea · Matrices, transformations & Markov chains
Can letter transitions tell languages apart? Markov chains for text
Research question
If a text is modelled as a Markov chain of letters, how different are the transition matrices of two languages, and can they be used to identify the language of a short passage?
Make it yours, then agree it with your teacher.
Next step: plan this IA free — the planner opens with this question filled in. For feedback on your own draft, the IA package (€39 once) gives you 5 AI feedback credits and every annotated exemplar in full.
Why it makes a good exploration
It uses real texts, counts you make yourself and a clear test (identify the language). The 'generated text' from your matrix is also a lively way to show what the model captures.
The mathematics you'll need
- Transition matrices from letter-pair counts
- Probability of a passage under each model (products, then logarithms)
- Comparing models with a likelihood ratio
- Success rate against passage length
- Generating text from the chain
Modelling the data, step by step
This idea compares models fitted to data. See it worked step by step, with a criterion tip at every step: Choosing and comparing models.
Model your own data Paste it from Desmos, GeoGebra or a spreadsheet and get the same play-by-play with your numbers. New to modelling? Start with the modelling workflow. Writing it up? The IA modelling planner comments on each paragraph as you draft — it never writes it for you.
Where the data comes from
Use public-domain books in two languages from Project Gutenberg.
- Project Gutenberg — 70,000+ free public-domain books as plain text — ideal for letter and word frequency counts.
Real data for this idea
- Primary source: Project Gutenberg: Free ebooks. Download two or three books as “Plain Text UTF-8” by hand and remove the licence header before counting. Licence: Public domain in the US; download a few books by hand (the site blocks bulk robots).
Links checked October 2026; we link to the data and never copy it. More free data sets.
A possible outline
- Explain the model and build two matrices.
- Generate sample text and comment on it.
- Classify test passages by likelihood.
- Measure accuracy against passage length.
- Reflect on unseen letter pairs and on two-letter memory.
Pitfalls that cost marks
- Multiplying tiny probabilities without logs.
- Zero probabilities for unseen pairs; explain how you handle them.
- Testing on the same text you trained on.
Showing personal engagement
- Use languages you speak.
- Find the shortest passage you can classify reliably.
- Try two dialects or two authors.
See Criterion C: personal engagement for what examiners look for.
Which course is it for?
| Course | Fit | Maths to lean on |
|---|---|---|
| AA SL | Not a natural fit | The core technique sits in the AI course or at HL; an AA SL student could use it only as clearly explained new mathematics. |
| AA HL | Good fit | Transition matrices from letter-pair counts; Probability of a passage under each model (products, then logarithms) |
| AI SL | Good fit | Transition matrices from letter-pair counts; Probability of a passage under each model (products, then logarithms) |
| AI HL | Good fit | Transition matrices from letter-pair counts; Probability of a passage under each model (products, then logarithms) |
Level: Solid. Needs some independent work beyond class examples. See how the IA differs between AA and AI, SL and HL.
How this idea reaches the top bands
Reflection (D)
For this idea, start with the weak points: multiplying tiny probabilities without logs; and zero probabilities for unseen pairs; explain how you handle them. Say how each one affects your answer.
Use of mathematics (E)
SL: transition matrices from letter-pair counts; probability of a passage under each model (products, then logarithms); comparing models with a likelihood ratio; success rate against passage length; generating text from the chain, each explained in your own words and carried out correctly.
How matrices & markov explorations reach the top bands in general: Matrices, transformations & Markov chains IA ideas. Guides: A · B · C · D · E · self-grade sheet.
Taking it further
Use pairs of letters as states and see whether accuracy improves.
Extending it for HL
Add an HL technique the question genuinely needs: what an HL matrices & markov exploration adds.
See a complete IA, marked
How long does a game of Snakes and Ladders last on my grandmother's board? (AI HL) is a complete matrices & markov exploration, marked on every criterion.
Frequently asked questions
Which IB Maths course is this IA idea for?
It works for AI HL, AA HL and AI SL. We rate it Solid: it needs some independent work beyond class examples.
What mathematics does this IA use?
Transition matrices from letter-pair counts, probability of a passage under each model (products, then logarithms), comparing models with a likelihood ratio, success rate against passage length and generating text from the chain.
Where can I get data for this IA?
Use public-domain books in two languages from Project Gutenberg. Checked sources and tools: Project Gutenberg.
Turn this idea into your IA
The planner opens with “Can letter transitions tell languages apart? Markov chains for text” filled in: the question, the maths and the data sources.
Your plan is kept in this browser. A free account keeps it with your drafts on any device: Join free · Sign in
Also: free IA self-check · A–E checklist (PDF) · have a draft? AI feedback or a review with Pete (€100)
Writing it up: how to write a Maths IA, step by step · structure and format · mistakes that cost marks · IA modelling, step by step
Similar ideas
- Ranking teams that haven't all played each other: the Colley matrixAI HLAA HLAmbitious
- Gambler's ruin: how long until someone runs out?AI HLAA HLAmbitious
- Does rain today predict rain tomorrow? A Markov chain for the weatherAI HLAA HLSolid
- Will one phone brand take over? Brand loyalty as a Markov chainAI HLSolid
All matrices & markov ideas · AI HL ideas · AA HL ideas · AI SL ideas · All 239 IA ideas