Updated · By Pete Bromfield, former Head of Maths with IB examining experience

IA idea · Matrices, transformations & Markov chains

Can letter transitions tell languages apart? Markov chains for text

AI HLAA HLAI SL Solid Also in: Probability, Statistics

Research question

If a text is modelled as a Markov chain of letters, how different are the transition matrices of two languages, and can they be used to identify the language of a short passage?

Make it yours, then agree it with your teacher.

Next step: plan this IA free — the planner opens with this question filled in. For feedback on your own draft, the IA package (€39 once) gives you 5 AI feedback credits and every annotated exemplar in full.

Why it makes a good exploration

It uses real texts, counts you make yourself and a clear test (identify the language). The 'generated text' from your matrix is also a lively way to show what the model captures.

The mathematics you'll need

  • Transition matrices from letter-pair counts
  • Probability of a passage under each model (products, then logarithms)
  • Comparing models with a likelihood ratio
  • Success rate against passage length
  • Generating text from the chain

Where the data comes from

Use public-domain books in two languages from Project Gutenberg.

  • Project Gutenberg — 70,000+ free public-domain books as plain text — ideal for letter and word frequency counts.

Real data for this idea

  1. Primary source: Project Gutenberg: Free ebooks. Download two or three books as “Plain Text UTF-8” by hand and remove the licence header before counting. Licence: Public domain in the US; download a few books by hand (the site blocks bulk robots).

Links checked October 2026; we link to the data and never copy it. More free data sets.

A possible outline

  1. Explain the model and build two matrices.
  2. Generate sample text and comment on it.
  3. Classify test passages by likelihood.
  4. Measure accuracy against passage length.
  5. Reflect on unseen letter pairs and on two-letter memory.

Pitfalls that cost marks

  • Multiplying tiny probabilities without logs.
  • Zero probabilities for unseen pairs; explain how you handle them.
  • Testing on the same text you trained on.

Showing personal engagement

  • Use languages you speak.
  • Find the shortest passage you can classify reliably.
  • Try two dialects or two authors.

See Criterion C: personal engagement for what examiners look for.

Which course is it for?

CourseFitMaths to lean on
AA SLNot a natural fitThe core technique sits in the AI course or at HL; an AA SL student could use it only as clearly explained new mathematics.
AA HLGood fitTransition matrices from letter-pair counts; Probability of a passage under each model (products, then logarithms)
AI SLGood fitTransition matrices from letter-pair counts; Probability of a passage under each model (products, then logarithms)
AI HLGood fitTransition matrices from letter-pair counts; Probability of a passage under each model (products, then logarithms)

Level: Solid. Needs some independent work beyond class examples. See how the IA differs between AA and AI, SL and HL.

How this idea reaches the top bands

Reflection (D)

For this idea, start with the weak points: multiplying tiny probabilities without logs; and zero probabilities for unseen pairs; explain how you handle them. Say how each one affects your answer.

Use of mathematics (E)

SL: transition matrices from letter-pair counts; probability of a passage under each model (products, then logarithms); comparing models with a likelihood ratio; success rate against passage length; generating text from the chain, each explained in your own words and carried out correctly.

How matrices & markov explorations reach the top bands in general: Matrices, transformations & Markov chains IA ideas. Guides: A · B · C · D · E · self-grade sheet.

Taking it further

Use pairs of letters as states and see whether accuracy improves.

Extending it for HL

Add an HL technique the question genuinely needs: what an HL matrices & markov exploration adds.

Frequently asked questions

Which IB Maths course is this IA idea for?

It works for AI HL, AA HL and AI SL. We rate it Solid: it needs some independent work beyond class examples.

What mathematics does this IA use?

Transition matrices from letter-pair counts, probability of a passage under each model (products, then logarithms), comparing models with a likelihood ratio, success rate against passage length and generating text from the chain.

Where can I get data for this IA?

Use public-domain books in two languages from Project Gutenberg. Checked sources and tools: Project Gutenberg.

Turn this idea into your IA

The planner opens with “Can letter transitions tell languages apart? Markov chains for text” filled in: the question, the maths and the data sources.

Plan this IA free

Your plan is kept in this browser. A free account keeps it with your drafts on any device: Join free · Sign in

Also: free IA self-check · A–E checklist (PDF) · have a draft? AI feedback or a review with Pete (€100)

Writing it up: how to write a Maths IA, step by step · structure and format · mistakes that cost marks · IA modelling, step by step

Similar ideas

All matrices & markov ideas · AI HL ideas · AA HL ideas · AI SL ideas · All 239 IA ideas