How we compared SQE1 practice questions

This page supports our article SQE1 question banks compared.

Declaration of interest. Solon Prep publishes SQE1 practice books and is one of the providers assessed below. We publish enough detail for the comparison to be tested and repeated.

1. Sources and dates

All material was tested in September 2026. For other providers we used each provider's free tier, free preview or public web pages. We bought one item for this test: the Revise SQE FLK1 practice assessment book, purchased on Amazon. All other questions were freely available on the providers' own websites without payment. Where a free tier was capped, we used the questions it unlocked.

Not reviewed:

• The University of Law: its free questions are app-only.
• TheOneHundred: its free mock could not be read.

2. Sample sizes

See section 5. A broken key makes a question look harder than it is, so other providers' panel averages exclude items whose key we flagged as wrong or arguable.

3. How questions were sampled

• Other providers: every free question available, up to about 60 per provider, taken in the order presented. None was selected by difficulty or topic.
• Solon: 120 questions drawn at random from the three published books using a fixed seed (random.seed(7)). The draw was fixed before any checking began: 60 from Essential SQE1 Questions and 30 from each Mock Papers book.
• We did not re-sample, drop rows, or re-run checks after seeing the results.

4. How accuracy was checked: the three-way rule

Every published error had to clear three separate checks. Each check tested the key and explanation against current primary sources, including legislation.gov.uk, BAILII, the Supreme Court website, the CPR and the SRA Standards and Regulations. If the checks split, the item was not counted.

Other providers

1. The first check answered each question blind, then tested the key and explanation against the primary sources.
2. Every flagged item then went through two further checks with the same neutral instructions. Those checks saw the question, the provider's key and the explanation, but not the first finding.

Solon's books. The same three checks were used, in the reverse order.

1. All 120 questions went through the first two checks with the same neutral instructions.
2. The final check reviewed 41 questions blind, using the same rubric: every explanation that either earlier check had flagged, plus 16 unflagged decoys. It was not told which were which.

Publication rule (both groups): an error counts only if all three checks flag it. Items where the checks split, or where a key was called "arguable" rather than wrong, are not counted. Cases where a second answer is defensible are recorded internally as judgement calls and are not counted in the published results.

Technical note for reproducibility: the three checks used Claude for one pass and two separate GPT-6-Sol runs for the other passes, with live web access for primary-source verification. The order was reversed for Solon's books as set out above.

Definitions

• Wrong answer key: the keyed option is not the best answer under the law of England and Wales at the test date.
• Explanation error: the key is right, but the explanation states the law wrongly. That means a wrong rule, a provision that doesn't exist, or superseded law presented as current.

5. How difficulty was measured: the panel

• Testing panel: six independently run systems. The same panel scored the SRA samples, every provider and Solon.
• Procedure: each panel member answers each question closed-book, twice, with the options shuffled between runs, giving 12 votes per question.
• Score: correct votes ÷ 2, giving a score from 0 to 6. Lower is harder. The bands are:
  • hard: 2.5 or below;
  • standard: 3 to 4.5;
  • easy: 5 or above.
• Clean scores: other providers' averages exclude items with a flagged key. On all items, QLTS School scored 3.19 rather than 3.31. Solon had no flagged keys, so its clean and all-items scores are the same.
• Calibration against real candidates: on the SRA's 130 pretested questions, panel score correlates with the percentage of candidates answering correctly at r = 0.36.
  • SRA questions the panel rated hard were answered correctly by 49% of candidates on average; standard, 54%; easy, 61%.
  • Individual panel members scored 36–57% on that set, against a real candidate average of 53%.
• What this means: the panel has a modest relationship with candidate performance and is useful for comparing question sets. It does not predict an individual candidate's result. "Level with the SRA's sample questions on our panel" is not a claim that any set is as hard as the real exam.

Technical note for reproducibility: the panel comprised Llama 3.1 8B, Phi-4, Gemma 3 12B, Mistral Small 24B, Qwen3 30B and GPT-OSS 20B.

Solon results in detail

• Solon's books print a difficulty label on each question. On the panel, questions labelled hard scored 1.5–2.0, standard 3.3 and easy 5.1–5.4.
• A fresh re-run of the panel matched the scores already stored for these questions at r = 0.72 (the mock papers) and r = 0.87 (Essential).

6. Statistics definitions

All of these were computed by script from the question text. Solon's figures cover all 1,520 questions in the three books.

• Wrong options explained: the share of questions in which every wrong option has its own explanation. Solon's books: 1,520 of 1,520.
• Stem length: words in the scenario plus the question. SRA pretested average: 137. Solon's books: 121.
• Key is the longest option: SRA 19%; Solon's books 10%.
• Absolute-word cue: the share of questions in which a word such as always, never, only, must, cannot, every, all, any or no appears in at least one wrong option but not in the key. SRA 36%; Solon's books 47%.
• "Best describes" lead-in: SRA 8%; Solon's books 50%.
• Personal names: the share of stems naming a person. SRA: 0 of 220.
• Application, not recall: the share of questions that need the law applied to facts rather than a rule recalled. This is the checker's judgement. SRA: 82%.

7. Limits

1. Mostly free material, for other providers. These are free tiers, previews, lead magnets and blog questions, plus the one book we bought (Revise SQE's FLK1 practice assessment book), and may differ from a provider's paid bank, for better or worse. Nothing here is a finding about any provider's full product.
2. Solon figures are for the published books only. Newer questions written for Solon's online question bank measured easier on the same panel (small samples: 3.85 and 5.15). None of the book figures applies to them.
3. Small samples. The College of Legal Practice (11), SQEMCQ (11 original, 3 keyed) and Revise SQE (12) are too small to rank. Law Drills' free tier unlocks only two subjects, and BARBRI's preview only four.
4. Three-check threshold. Every check used live verification against primary sources. The three-way publication rule is deliberately conservative, and the right of reply provides a further route for challenge and correction.
5. Self-review. One checking system belongs to the same family used to rebuild Solon's books, and the other is also used in Solon's question-writing checks. Other providers had no equivalent conflict, so we disclose it here and applied the same all-three-must-agree threshold throughout.
6. How much the third check filtered. For Solon, the first two checks flagged 10 explanations. The blind final check upheld 1 and judged the other 9 accurate, most of them objections to wording the earlier checks found too absolute. For other providers, the two later checks confirmed or rejected items flagged by the first. The publication rule is the same in both directions.
7. The panel has a modest relationship with candidate performance (r = 0.36; see section 5). Solon's lead over QLTS School and The College of Legal Practice is within the panel's margin of error.
8. Content changes. Providers may have changed their questions since we tested them. Each finding is dated to September 2026. We re-checked the cited items against the live pages before publishing, where we could still reach them.

8. Corrections and right of reply

If you're a provider and believe we've got something wrong, contact us with the item and your reasoning. We'll review it and, where we're wrong, correct the article and this page within 5 working days, recording the change below.

Correction log