IA idea · Matrices, transformations & Markov chains

Can letter transitions tell languages apart? Markov chains for text

AI HLAA HLAI SL Solid Also in: Probability, Statistics

Research question

If a text is modelled as a Markov chain of letters, how different are the transition matrices of two languages, and can they be used to identify the language of a short passage?

Adapt it: change the place, the data or the comparison until the question is yours.

Free: the A–E checklist an examiner uses, by email ↓

Why it makes a good exploration

It uses real texts, counts you make yourself and a clear test (identify the language). The 'generated text' from your matrix is also a lively way to show what the model captures.

The mathematics you'll need

Transition matrices from letter-pair counts
Probability of a passage under each model (products, then logarithms)
Comparing models with a likelihood ratio
Success rate against passage length
Generating text from the chain

Course labels show where a technique sits; using maths from outside your course is fine if you explain it clearly and say it is new to you.

Modelling the data, step by step

This idea compares models fitted to data. See it worked step by step, with a criterion tip at every step: Choosing and comparing models.

Model your own data Paste it from Desmos, GeoGebra or a spreadsheet and get the same play-by-play with your numbers. New to modelling? Start with the modelling workflow. Writing it up? The IA modelling planner comments on each paragraph as you draft — it never writes it for you.

Where the data comes from

Use public-domain books in two languages from Project Gutenberg.

Project Gutenberg — 70,000+ free public-domain books as plain text — ideal for letter and word frequency counts.

Cite every source in a footnote where you use it and in your bibliography. Check the licence of any dataset you download.

A possible outline

Explain the model and build two matrices.
Generate sample text and comment on it.
Classify test passages by likelihood.
Measure accuracy against passage length.
Reflect on unseen letter pairs and on two-letter memory.

Pitfalls that cost marks

Multiplying tiny probabilities without logs.
Zero probabilities for unseen pairs; explain how you handle them.
Testing on the same text you trained on.

Showing personal engagement

Use languages you speak.
Find the shortest passage you can classify reliably.
Try two dialects or two authors.

See Criterion C: personal engagement for what examiners look for.

Which course is it for?

Course	Fit	Maths to lean on
AA SL	Not a natural fit	The core technique sits in the AI course or at HL; an AA SL student could use it only as clearly explained new mathematics.
AA HL	Good fit	Transition matrices from letter-pair counts; Probability of a passage under each model (products, then logarithms)
AI SL	Good fit	Transition matrices from letter-pair counts; Probability of a passage under each model (products, then logarithms)
AI HL	Good fit	Transition matrices from letter-pair counts; Probability of a passage under each model (products, then logarithms)

Level: Solid. Needs some independent work beyond class examples. See how the IA differs between AA and AI, SL and HL.

How this idea reaches the top bands

Personal engagement (C)

Collect the data for your matrix yourself (counting transitions, measuring a shape), and choose the states or the transformation from a situation you care about.

Reflection (D)

Question the model's assumptions: is the process memoryless, are the probabilities constant, does the transformation preserve what it should? Say how each affects your conclusion. For this idea, start with: multiplying tiny probabilities without logs — say how it affects your answer.

Use of mathematics (E)

SL: Matrices are in AI HL. At SL, keep to small matrices you explain carefully as new mathematics, with every multiplication shown once and the result interpreted.

HL: Transition matrices, powers, steady states and eigenvalues used correctly, with diagonalisation or a general result derived, and the long-run behaviour interpreted in context.

Criteria A and B (presentation and communication) work the same way for every idea: see the guides to Criterion A and Criterion B.

Taking it further

Use pairs of letters as states and see whether accuracy improves.

Extending it for HL

Diagonalise the matrix to find a formula for the nth state, or compare the steady state with what your data actually shows.

See a complete IA, marked

Our annotated exemplar How long does a game of Snakes and Ladders last on my grandmother's board? (AI HL) asks a different question, but shows how a complete matrices & markov exploration is structured and marked, with an examiner's comment on every criterion. Free excerpts and the full marking table are on its page.

Before you start: the checklist an examiner uses

Every check for Criteria A–E in a 4-page PDF, the mistakes that cost the most marks and a self-assessment grid. We'll email it with a short IA tip every few days, timed to your deadline if you give it. Free — no account, no payment.

Get the free checklist

Turn this idea into your IA

Plan this IAOpens the planner with this question, the maths and data sources filled in. Criteria checklistCheck your plan or draft against the A–E descriptors and self-grade before you hand it in. AI feedback on a draftCriterion-by-criterion feedback with evidence from your own text. Coaching, not rewriting. Examiner reviewPete, an IB examiner, reads your draft and talks it through with you (€100).