Building a CAT Test with Guessing Parameter (3PL IRT) in Concerto Platform

Prerequisites

Before starting this tutorial, make sure you have:

• Concerto Platform running via Docker (docker-compose up -d)
• Access to http://localhost/login and logged in as admin
• Read the CAT Theory document to understand IRT concepts

Concerto Platform — Known Rules (From Experience)

Before building, understand these important rules discovered through testing:

Concept: What is the 3PL Model (with Guessing)?

The 3-Parameter Logistic (3PL) model extends basic IRT by adding a guessing parameter (c) [2], [5]:

Where:

• $\theta$ (theta) — test-taker's ability estimate (starts at 0)
• $a$ — discrimination: how well the item separates ability levels
• $b$ — difficulty: the ability level at which $P = 0.5$ (ignoring guessing)
• $c$ — guessing: minimum probability of correct answer (e.g. 0.25 for 4-choice MCQ)

Why does guessing matter?

Without guessing (2PL), if theta is very low, $P(\text{correct}) \rightarrow 0$. But in reality, a test-taker can still guess correctly — especially on multiple choice. The guessing parameter $c$ sets a floor on the probability [2]:

Item Information with Guessing

The information provided by a 3PL item is [3], [5]:

Items with high discrimination ($a$) and difficulty near current theta ($b \approx \theta$) provide the most information [3], [14].

Concept: Bayesian EAP Theta Estimation

This tutorial uses Bayesian Expected A Posteriori (EAP) estimation — the most robust method for CAT theta estimation [8], [10].

Why Bayesian EAP instead of Newton-Raphson?

How EAP Works

EAP combines the likelihood of the observed responses with a prior distribution (our belief about ability before the test) [8]:

Where:

• $L(\theta \mid \mathbf{u}) = \prod_{j=1}^{n} P_j(\theta)^{u_j} [1-P_j(\theta)]^{1-u_j}$ — likelihood of all responses
• $\pi(\theta) = \mathcal{N}(0, 1)$ — standard normal prior (most people have average ability)
• $\mathbf{u} = (u_1, \ldots, u_n)$ — vector of responses (1=correct, 0=incorrect)

Numerical Approximation (used in our R code)

The integral is approximated using Gauss-Hermite quadrature — evaluating the integrand at a grid of $K$ theta points $\{\theta_k\}$ with weights $\{w_k\}$ [6], [8]:

The Standard Error is also computed from the posterior variance:

Part 1: Create the Item Bank (Data Table)

Concept

The item bank stores all test questions along with their IRT parameters. The 3PL model requires three parameters per item: discrimination ($a$), difficulty ($b$), and guessing ($c$) [2], [5], [15].

Steps

1. Click Data Tables in the left menu → Add new
2. Name it item_bank_3pl
3. Add these columns by clicking Add column:

4. Click Save
5. Click Edit data and add sample items with varying difficulty:

6. Click Save

Note: All guessing values are 0.25 because these are 4-choice MCQ items. The probability of guessing correctly = 1/4 = 0.25.

Part 2: Create the Test

1. Click Tests → Add new → name it cat_3pl_test → Save
2. Click the Test flow tab
3. You will see test start and test end on the canvas

Part 3: Build the Test Flow

Final Flow Overview

[test start]
     ↓
[eval - init]           Initialize all variables + response history
     ↓
[eval - select item] ←──────────────────────────┐
     ↓                                           │
[showPage - question]   Show item to user        │
     ↓                                           │
[eval - score]          Score + Bayesian EAP     │
     ↓                                           │
[if]                    Test complete?           │
  │ false ──────────────────────────────────────┘
  │ true
  ↓
[eval - compute result] Compute labels + SE
     ↓
[showPage - result]     Show final score + SE
     ↓
[test end]

Node 1: eval - init — Initialize Variables

Concept

This node sets all starting values before the test begins. For Bayesian EAP, we also initialize:

• responses — a vector tracking all responses (1=correct, 0=incorrect) across items
• items_a, items_b, items_c — vectors tracking IRT parameters of answered items
• se_theta — the standard error of the theta estimate
• theta = 0 — prior mean (start at average ability) [12]

The response history vectors are essential for EAP because it needs all previous responses and item parameters to compute the posterior, not just the most recent one [8].

Steps

1. Right-click canvas → eval
2. Rename it to eval - init
3. Click the node → edit Code field → paste:

# ── Ability estimate ──────────────────────────────────────────────────────────
# Start at population mean θ = 0 (prior mean for Bayesian EAP)
theta    <- 0

# ── Standard error of theta estimate ─────────────────────────────────────────
# Starts high (very uncertain), decreases as more items are answered
se_theta <- 999

# ── Test control variables ────────────────────────────────────────────────────
answered  <- 0      # number of items answered so far
max_items <- 10     # fixed-length stopping rule

# ── Used item tracking ────────────────────────────────────────────────────────
# Prevents the same item from being shown twice
used_items <- c()

# ── Response history — required for Bayesian EAP ─────────────────────────────
# responses: 1 = correct, 0 = incorrect, one entry per answered item
responses <- c()

# IRT parameter history — one entry per answered item (same order as responses)
items_a   <- c()   # discrimination parameters of answered items
items_b   <- c()   # difficulty parameters of answered items
items_c   <- c()   # guessing parameters of answered items

# ── Question display variables ────────────────────────────────────────────────
correct_answer <- ""
question       <- ""
option_a       <- ""
option_b       <- ""
option_c       <- ""
option_d       <- ""
current_id     <- 0

# ── Scoring totals ────────────────────────────────────────────────────────────
total_correct <- 0
test_complete <- FALSE

4. Click Save

Add output ports (↑)

Click red + for each variable. Then click each port → check Flow variable pointer → set Pointed variable name to the same name → Save:

• theta
• se_theta
• answered
• max_items
• used_items
• responses
• items_a
• items_b
• items_c
• correct_answer
• question
• option_a
• option_b
• option_c
• option_d
• current_id
• total_correct
• test_complete

Each should show a ↑ arrow when done.

Connect

Drag from test start out → eval - init in

Details

The eval - init node runs exactly once — at the very beginning of the test, immediately after test start. Its sole purpose is to initialize every variable that will be used throughout the entire CAT session.

It is the simplest node in the flow but also the most foundational: if any variable is missing or wrongly typed here, every subsequent node will fail.

Complete R Code

# ── Ability estimate ──────────────────────────────────────────────────────────
# Start at population mean θ = 0 (prior mean for Bayesian EAP)
theta    <- 0

# ── Standard error of theta estimate ─────────────────────────────────────────
# Starts high (very uncertain), decreases as more items are answered
se_theta <- 999

# ── Test control variables ────────────────────────────────────────────────────
answered  <- 0      # number of items answered so far
max_items <- 10     # fixed-length stopping rule

# ── Used item tracking ────────────────────────────────────────────────────────
# Prevents the same item from being shown twice
used_items <- numeric(0)

# ── Response history — required for Bayesian EAP ─────────────────────────────
# responses: 1 = correct, 0 = incorrect, one entry per answered item
responses <- numeric(0)

# IRT parameter history — one entry per answered item (same order as responses)
items_a   <- numeric(0)   # discrimination parameters of answered items
items_b   <- numeric(0)   # difficulty parameters of answered items
items_c   <- numeric(0)   # guessing parameters of answered items

# ── Question display variables ────────────────────────────────────────────────
correct_answer <- ""
question       <- ""
option_a       <- ""
option_b       <- ""
option_c       <- ""
option_d       <- ""
current_id     <- 0

# ── Scoring totals ────────────────────────────────────────────────────────────
total_correct <- 0
test_complete <- FALSE

Variable-by-Variable Explanation

theta <- 0

What it is: The current ability estimate $\hat{\theta}$ — the CAT's best guess of the test-taker's true latent ability on the IRT scale.

theta <- 0

Why start at 0: In IRT, the ability scale is standardized so that the population mean is 0 and the standard deviation is 1. Starting at $\hat{\theta}_0 = 0$ means we begin with the assumption that the test-taker has average ability — the most neutral and statistically justified starting point before any evidence is collected.

This is the prior mean of the Bayesian EAP estimation. The standard normal prior $\pi(\theta) = \mathcal{N}(0, 1)$ used in eval - score is centered at 0, so starting theta at 0 is consistent with the prior.

What happens if we started elsewhere:

How theta evolves:

Start:    theta = 0.0000  (prior mean, no data)
After Q1: theta = 0.3821  (answered correctly — ability estimate rises)
After Q2: theta = 0.1504  (answered incorrectly — estimate falls)
After Q3: theta = 0.2813  (answered correctly — rises again)
...
After Q10: theta = 0.4217  (converged estimate)

The EAP algorithm in eval - score updates this value after every response.

se_theta <- 999

What it is: The Standard Error of the theta estimate — a measure of how precisely we know the test-taker's ability:

se_theta <- 999

Why 999 and not 0, NA, or Inf:

Before any items are answered ($n = 0$), the true SE is mathematically infinite:

We cannot use Inf directly because Concerto may corrupt infinite values during flow variable serialization. 999 is a sentinel value — a deliberately large finite number that means "infinitely uncertain":

Why it matters for the stopping rule: In eval - score, the SE-based stopping rule is:

test_complete <- (se_theta < 0.3) | (answered >= max_items)

With se_theta = 999 initially:

999 < 0.3   # → FALSE  ✅ test correctly does not stop before any items

If se_theta were 0:

0 < 0.3   # → TRUE  ❌ test would immediately end with zero items answered

How SE evolves:

SE is a real EAP-computed value from item 1 onward — 999 exists only for the single moment before the first item is shown.

answered <- 0

What it is: A counter tracking how many items have been answered so far.

answered <- 0

Why start at 0: No items have been answered at test start. This counter increments by 1 in eval - score after each submission:

# In eval - score:
answered <- as.numeric(answered) + 1

How it is used:

Why integer not logical: answered participates in arithmetic (answered + 1) and comparison (answered >= max_items), so it must be numeric. Starting as 0 (integer) ensures this.

max_items <- 10

What it is: The fixed-length stopping rule threshold — the maximum number of items the test will administer.

max_items <- 10

Why 10: 10 is a reasonable default for a demonstration CAT. In operational CATs:

To change the test length: Simply change this one value:

max_items <- 5    # short demo
max_items <- 20   # standard test
max_items <- 30   # longer assessment

How it flows through the system:

eval - init:     max_items = 10  (set here)
      ↓ output ↑
eval - select item: receives max_items, passes to showPage
      ↓
showPage - question: displays "Question X of 10"
      ↓
eval - score:    max_items received, used in:
                 test_complete <- n_answered >= max_items

Why defined here and not hardcoded in eval - score: Centralizing configuration in eval - init means you only need to change one place to adjust test length. If it were hardcoded in eval - score, you would need to update two nodes whenever you changed the test length.

used_items <- numeric(0)

What it is: A vector tracking the IDs of all items that have already been administered. Used in eval - select item to exclude already-seen items from selection.

used_items <- numeric(0)

Why numeric(0) and not c() or NULL:

Using numeric(0) ensures that when we later do c(used_items, current_id) where current_id is an integer, the result is always a numeric vector — never a character or list.

How it is used in eval - select item:

if (length(used_items) > 0) {
  items <- items[!items$id %in% used_items, ]
}

On the first call: length(numeric(0)) = 0 → condition is FALSE → filter skipped → all items available.

Progression across the test:

# Before Q1:  numeric(0)         → all 10 items available
# Before Q2:  c(5)               → 9 items available
# Before Q3:  c(5, 6)            → 8 items available
# Before Q10: c(5,6,3,7,1,9,4,8,10) → 1 item remaining

responses <- numeric(0)

What it is: A vector storing the binary response history — one entry per answered item:

• 1 = correct
• 0 = incorrect

responses <- numeric(0)

This is the $\mathbf{u} = (u_1, u_2, \ldots, u_n)$ vector in the EAP likelihood formula:

Why this is critical for Bayesian EAP: Unlike Newton-Raphson (which uses only the most recent item), EAP recomputes theta from scratch after every item using all previous responses. Without this vector, EAP cannot function — it needs the complete response history to compute the likelihood.

Why numeric(0) not c(): Same reasoning as used_items — explicit numeric type prevents type errors when arithmetic is performed on elements in the EAP loop.

Progression across the test:

# Before Q1:  numeric(0)      → no responses yet
# After Q1:   c(0)            → answered incorrectly
# After Q2:   c(0, 1)         → Q2 correct
# After Q3:   c(0, 1, 1)      → Q3 correct
# After Q10:  c(0,1,1,0,1,0,1,1,0,1) → full history

In eval - score, the new response is appended:

responses <- c(responses, is_correct)

items_a <- numeric(0), items_b <- numeric(0), items_c <- numeric(0)

What they are: Three parallel vectors that store the IRT parameters of every item that has been answered, in the same order as responses.

items_a <- numeric(0)   # discrimination parameters of answered items
items_b <- numeric(0)   # difficulty parameters of answered items
items_c <- numeric(0)   # guessing parameters of answered items

Why three separate vectors: The EAP inner loop in eval - score iterates over answered items and needs all three parameters for each:

for (j in seq_len(n_answered)) {
  Pj <- items_c[j] + (1 - items_c[j]) / (1 + exp(-items_a[j] * (th - items_b[j])))
  ...
}

Storing them as separate parallel numeric vectors (rather than a data frame or list) makes element-wise access inside the loop reliable and avoids type coercion issues.

Why numeric(0) not c(): The EAP code does arithmetic on these vectors (1 - items_c[j], -items_a[j] * ...). Initializing as numeric(0) guarantees they are numeric before any elements are appended.

Parallel structure — they must always have the same length:

# After 3 items (responses = c(0, 1, 1)):
items_a <- c(0.8, 1.5, 1.4)    # item 1: a=0.8, item 2: a=1.5, item 3: a=1.4
items_b <- c(-2.0, 0.0, 0.5)   # item 1: b=-2.0, etc.
items_c <- c(0.25, 0.25, 0.25)  # all 4-choice MCQ so c=0.25 for all

# Invariant: length(responses) == length(items_a) == length(items_b) == length(items_c)

In eval - score, all four vectors are appended together in the same block:

responses <- c(responses, is_correct)
items_a   <- c(items_a,   a_new)
items_b   <- c(items_b,   b_new)
items_c   <- c(items_c,   c_new)

This guarantees the parallel structure is maintained across all iterations.

Question Display Variables

What they are: Placeholder variables for the current item's content. These are populated by eval - select item before each question is shown.

correct_answer <- ""
question       <- ""
option_a       <- ""
option_b       <- ""
option_c       <- ""
option_d       <- ""
current_id     <- 0

Why initialize as empty strings: These variables must exist as flow variables from the very first node. If they were not initialized here, eval - select item might receive NULL on the first iteration, causing errors during type conversion or string operations.

Why current_id <- 0 (not numeric(0)): current_id is used in a SQL WHERE clause in eval - score:

paste0("SELECT * FROM item_bank_3pl WHERE id = ", current_id)

An ID of 0 produces valid (if harmless) SQL: WHERE id = 0. This will return zero rows — safely handled. By contrast, numeric(0) would produce WHERE id = (incomplete SQL — an error).

Flow of display variables:

eval - init:          question = ""    (blank placeholder)
      ↓
eval - select item:   question = "What is 7x8?"   (filled by item selection)
      ↓
showPage - question:  {{question}} → "What is 7x8?"   (displayed to user)
      ↓
eval - score:         correct_answer used for scoring
      ↓
eval - select item:   question = "What is √169?"  (next item, overwrites)
      ↓
showPage - question:  {{question}} → "What is √169?"

total_correct <- 0

What it is: A running count of how many items the test-taker answered correctly.

total_correct <- 0

How it is updated in eval - score:

is_correct    <- as.integer(answer == correct_answer)
total_correct <- as.numeric(total_correct) + is_correct

How it is used in eval - compute result:

pct_correct <- round((total_correct / answered) * 100, 1)

Why this is separate from theta: total_correct is a classical test theory metric (raw score / percentage correct). Theta is an IRT metric. They measure the same construct differently:

Both are reported on the results page to give test-takers a familiar score alongside the IRT estimate.

test_complete <- FALSE

What it is: A logical flag that controls whether the if node routes to the next item or to the results page.

test_complete <- FALSE

Why initialize as FALSE: At test start, the test is obviously not complete. Initializing as FALSE means the if node expression test_complete evaluates correctly on the first pass without errors.

How it is updated in eval - score:

test_complete <- n_answered >= max_items

How the if node uses it:

test_complete = FALSE  →  if node "false" port  →  loop back to eval - select item
test_complete = TRUE   →  if node "true" port   →  go to eval - compute result

Why logical not integer: The if node expression field expects a logical (TRUE/FALSE) value. If test_complete were an integer (0 or 1), the if node might not evaluate it correctly. Initializing as FALSE (logical) and keeping it as logical throughout avoids this issue.

Why Every Variable Must Be Initialized Here

Concerto flow variable pointers only work for variables that already exist in the R session. If a variable is first created in eval - select item or eval - score (instead of eval - init), it will not be available as a flow variable in earlier rounds when the loop returns to that node.

The pattern is:

eval - init creates ALL variables with correct types
      ↓
eval - select item reads some, overwrites some, passes all forward
      ↓
showPage - question reads some, adds `answer`
      ↓
eval - score reads all, updates some, passes all forward
      ↓
if node routes based on test_complete
      ↓
loop back to eval - select item (which now has updated values from eval - score)

If any variable is missing from eval - init, the first time the loop returns to eval - select item, that variable will be NULL — causing either a silent error or a crash.

Type Choices — Why They Matter

Summary

eval - init is the configuration and initialization hub of the entire CAT session. Its design principles are:

1. Initialize everything — every variable used anywhere in the flow must be created here with the correct type
2. Use safe types — numeric(0) for empty vectors, FALSE for logicals, 0 for numeric counters
3. Use sentinel values — se_theta = 999 for "not yet computed" infinity
4. Centralize configuration — max_items = 10 here means one place to change test length
5. Export everything — every variable needs an output port with Flow variable pointer (↑)

Node 2: eval - select item — CAT Item Selection (3PL Maximum Information)

Concept

Selects the next item using the Maximum Information criterion based on the 3PL model [3], [11], [14].

Step 1: Compute $P_i(\hat{\theta})$ for each remaining item [2]: $P_i(\theta) = c_i + \frac{1 - c_i}{1 + e^{-a_i(\theta - b_i)}}$

Step 2: Compute $I_i(\hat{\theta})$ for each item [3], [5]: $I_i(\theta) = \frac{a_i^2 [P_i(\theta) - c_i]^2}{(1-c_i)^2} \cdot \frac{1 - P_i(\theta)}{P_i(\theta)}$

Step 3: Select $i^* = \underset{i \notin \mathcal{U}}{\arg\max}\ I_i(\hat{\theta})$ [11], [14]

Steps

1. Right-click canvas → eval
2. Rename it to eval - select item
3. Click the node → edit Code field → paste:

# ── Load all items from the item bank ─────────────────────────────────────────
items <- concerto.table.query("SELECT * FROM item_bank_3pl")

# ── Remove already-used items to prevent repetition ───────────────────────────
if (length(used_items) > 0) {
  items <- items[!items$id %in% used_items, ]
}

# ── Convert IRT parameter columns to numeric ───────────────────────────────────
items$difficulty     <- as.numeric(items$difficulty)
items$discrimination <- as.numeric(items$discrimination)
items$guessing       <- as.numeric(items$guessing)

# ── Extract IRT parameter vectors ─────────────────────────────────────────────
a <- items$discrimination
b <- items$difficulty
c <- items$guessing

# ── Step 1: Calculate P(θ) for each item using 3PL formula ───────────────────
# P(θ) = c + (1-c) / (1 + exp(-a*(θ-b)))
P <- c + (1 - c) / (1 + exp(-a * (theta - b)))

# ── Step 2: Calculate Item Information I(θ) ───────────────────────────────────
# I(θ) = a² * [(P-c)²/(1-c)²] * [(1-P)/P]
# Safety: prevent division by zero when P ≈ 0 or P ≈ 1
P_safe <- pmax(pmin(P, 0.9999), 0.0001)
items$information <- (a^2) * ((P_safe - c)^2 / (1 - c)^2) * ((1 - P_safe) / P_safe)

# ── Step 3: Select item with maximum information ───────────────────────────────
selected <- items[which.max(items$information), ]

# ── Set question display variables ────────────────────────────────────────────
question       <- as.character(selected$question)
option_a       <- as.character(selected$option_a)
option_b       <- as.character(selected$option_b)
option_c       <- as.character(selected$option_c)
option_d       <- as.character(selected$option_d)
correct_answer <- as.character(selected$correct_answer)
current_id     <- as.integer(selected$id)

# ── Track this item as used ───────────────────────────────────────────────────
used_items <- c(used_items, current_id)

4. Click Save

Add input ports (↓)

Click blue + for each. Click each port → check Flow variable pointer → Pointed variable name → Save:

• theta
• answered
• max_items
• used_items

Add output ports (↑)

Click red + for each. Click each port → check Flow variable pointer → Pointed variable name → Save:

• question
• option_a
• option_b
• option_c
• option_d
• correct_answer
• current_id
• used_items
• answered
• max_items
• theta

Connect

Drag from eval - init out → eval - select item in

Details

The eval - select item node runs at the start of every CAT loop iteration — once before each question is shown. It is the implementation of the CAT item selection algorithm: given the current ability estimate $\hat{\theta}$, it selects the most informative unused item from the bank.

It performs four sequential tasks:

Task 1: Load All Items from the Database

items <- concerto.table.query("SELECT * FROM item_bank_3pl")

What it does

Queries the entire item_bank_3pl table and returns all rows as an R data frame stored in items.

Why SELECT *

We need all columns — question text, options, correct answer, and IRT parameters — so SELECT * retrieves everything in one call. A more targeted query like SELECT id, discrimination, difficulty, guessing would be faster for large banks, but for typical CAT banks of 50–200 items the difference is negligible.

What items looks like after this line

Data types from Concerto database

Concerto returns all database values as character strings by default, regardless of the column type declared in the Data Table. This is why explicit type conversion is always required before any mathematical operations.

Task 2: Filter Out Already-Used Items

if (length(used_items) > 0) {
  items <- items[!items$id %in% used_items, ]
}

Mathematical notation

This implements the constraint $i \notin \mathcal{U}$ in the item selection rule:

Where $\mathcal{U}$ is the set of already-used item IDs.

Line-by-line breakdown

length(used_items) > 0

Checks whether any items have already been administered. On the very first question, used_items was initialized as numeric(0) (an empty vector) in eval - init, so length(numeric(0)) = 0 and the filter block is skipped entirely — all items remain available.

From question 2 onward, used_items grows by one ID per answered item, so the filter runs.

# Before question 1:
used_items <- numeric(0)
length(used_items)        # → 0  → skip filter

# Before question 2 (item 5 was used):
used_items <- c(5)
length(used_items)        # → 1  → run filter

items$id %in% used_items

The %in% operator checks membership. For each element of items$id, returns TRUE if that ID is in used_items, FALSE otherwise:

items$id    <- c(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)
used_items  <- c(5, 3, 7)

items$id %in% used_items
# → c(FALSE, FALSE, TRUE, FALSE, TRUE, FALSE, TRUE, FALSE, FALSE, FALSE)
#          1      2     3      4     5      6     7      8      9     10

Items 3, 5, and 7 were already used → TRUE.

!items$id %in% used_items

The ! operator flips all logical values — keeping items that are not in used_items:

!c(FALSE, FALSE, TRUE, FALSE, TRUE, FALSE, TRUE, FALSE, FALSE, FALSE)
# →  c(TRUE,  TRUE, FALSE, TRUE, FALSE,  TRUE, FALSE,  TRUE,  TRUE,  TRUE)

Items 1, 2, 4, 6, 8, 9, 10 remain → these are the candidates for selection.

items[..., ]

Subsets the data frame — keeping only rows where the logical vector is TRUE:

items <- items[c(TRUE, TRUE, FALSE, TRUE, FALSE, TRUE, FALSE, TRUE, TRUE, TRUE), ]
# Keeps rows for items 1, 2, 4, 6, 8, 9, 10
# Removes rows for items 3, 5, 7 (already used)

The trailing comma in items[condition, ] is required in R to indicate we are subsetting rows (not columns).

Progression across the test

Task 3: Type Conversion and Parameter Extraction

Convert IRT columns to numeric

items$difficulty     <- as.numeric(items$difficulty)
items$discrimination <- as.numeric(items$discrimination)
items$guessing       <- as.numeric(items$guessing)

As noted above, Concerto returns all database values as character strings. Without this conversion:

# What Concerto returns:
items$difficulty[1]      # → "0.5"   (a string)

# What happens without conversion:
0.5 - "0.5"              # → Error: non-numeric argument to binary operator

# What happens with conversion:
as.numeric("0.5") - 0.5  # → 0.0   ✅

Only the three IRT parameter columns need conversion here. The question text and options remain as character strings (which is correct for display).

Extract parameter vectors

a <- items$discrimination
b <- items$difficulty
c <- items$guessing

Pulls the three IRT parameter columns into standalone vectors a, b, c.

Why create separate vectors?

R is vectorized — arithmetic operations on vectors are applied element-wise across all elements simultaneously, without explicit loops:

a <- c(0.8, 1.0, 1.2, 1.5, 1.8)   # 5 items
b <- c(-2.0, -1.0, 0.0, 1.0, 2.0)
c <- c(0.25, 0.25, 0.25, 0.25, 0.25)
theta <- 0.0

# This computes P for ALL 5 items at once:
P <- c + (1 - c) / (1 + exp(-a * (theta - b)))
# → c(0.969, 0.938, 0.625, 0.312, 0.254)

This is equivalent to running a for-loop over all items but is:

• Faster — implemented in compiled C internally
• Cleaner — no loop boilerplate
• Less error-prone — no index management

Task 4: Calculate P(θ) for All Items

P <- c + (1 - c) / (1 + exp(-a * (theta - b)))

The 3PL Model

This implements the 3-Parameter Logistic (3PL) Item Characteristic Curve:

Where:

• $\theta$ — current ability estimate (scalar, same for all items)
• $a_i$ — discrimination parameter of item $i$
• $b_i$ — difficulty parameter of item $i$
• $c_i$ — guessing parameter of item $i$

Breaking down the formula component by component

Role of each parameter

Difficulty $b_i$ — where on the theta scale P = 0.5 + c/2

The difficulty parameter shifts the curve left or right along the theta axis. At $\theta = b_i$ (ignoring guessing, $c=0$):

With guessing ($c > 0$), the inflection point is slightly above $b_i$.

Low b (easy item):          High b (hard item):
P                           P
1.0 ──────────              1.0         ──────────
0.5 ──┐                     0.5               ──┐
0.25  │ (guessing)          0.25 (guessing)   │
      └────                             └────
      θ                                       θ
   b=-2.0                               b=+2.0

Discrimination $a_i$ — steepness of the curve

Higher $a_i$ means the curve rises more steeply — the item more sharply separates test-takers above vs below the difficulty level:

High a (steep):     Low a (shallow):
P                   P
1.0  ─┐             1.0    ──────────
0.5  ─┤             0.5   ─
0.25  │             0.25 ─
     ─┘
      θ=b            θ=b

Guessing $c_i$ — lower asymptote

Sets the floor on $P_i(\theta)$. Even at $\theta \to -\infty$, $P_i \to c_i$:

For 4-choice MCQ items: $c_i = 0.25$ (random guessing = 1/4).

Concrete example at $\theta = 0$

For 10 items with increasing difficulty ($b$ from -2.0 to +2.5):

At $\theta=0$, easy items have high $P$ (near 1), hard items approach the guessing floor (0.25).

Task 5: Calculate Item Information I(θ)

P_safe <- pmax(pmin(P, 0.9999), 0.0001)
items$information <- (a^2) * ((P_safe - c)^2 / (1 - c)^2) * ((1 - P_safe) / P_safe)

The 3PL Item Information Function (IIF)

The Item Information Function measures how much statistical information item $i$ provides about $\theta$ at a given ability level:

Why two lines? — Probability clamping

P_safe <- pmax(pmin(P, 0.9999), 0.0001)

The IIF formula has $P_i(\theta)$ in the denominator — division by $P_i$ becomes a problem when $P_i \approx 0$, and $(1 - P_i)/P_i \to \infty$. Similarly, the $(P_i - c_i)$ term is invalid if $P_i < c_i$ due to floating point errors.

pmin(P, 0.9999) — element-wise minimum: caps each $P_i$ at 0.9999:

pmin(c(0.9999, 1.0000, 0.8), 0.9999)
# → c(0.9999, 0.9999, 0.8)

pmax(..., 0.0001) — element-wise maximum: floors each result at 0.0001:

pmax(c(0.0001, 0.0000, 0.5), 0.0001)
# → c(0.0001, 0.0001, 0.5)

Note: pmax/pmin (lowercase p) operate element-wise on vectors, unlike max/min which return a single scalar. This is important here since P is a vector of 10 values.

Breaking down the IIF formula

Component 1: $a_i^2$ — Discrimination squared

Items with higher discrimination contribute quadratically more information. An item with $a=2.0$ provides $4\times$ more information (at its peak) than one with $a=1.0$.

a^2

Component 2: $\frac{(P_i - c_i)^2}{(1-c_i)^2}$ — Guessing correction

This term scales down the information to account for the guessing floor. It equals $\left(\frac{P_i - c_i}{1-c_i}\right)^2$, which is the proportion of the probability above the guessing floor relative to the maximum possible.

(P_safe - c)^2 / (1 - c)^2

At the difficulty point ($\theta = b_i$, where $P_i \approx 0.5 + c_i/2$):

So the guessing parameter reduces peak information by factor $(1-c_i)^2$. For $c=0.25$: reduction factor = $(0.75)^2 = 0.5625$ — guessing cuts peak information nearly in half.

Component 3: $\frac{1-P_i}{P_i}$ — Uncertainty term

This term is maximized when $P_i = 0.5$ and approaches 0 at both extremes:

(1 - P_safe) / P_safe

Combined with the guessing correction, the true maximum information for a 3PL item occurs slightly above the difficulty parameter $b_i$, not exactly at it, because the guessing floor shifts the optimal point upward.

Where is information maximized?

Information $I_i(\theta)$ is maximized where $b_i \approx \hat{\theta}$ — items work best when difficulty matches ability:

I(θ)
  │         ╭─────╮         ← Item with b=0.5 (peaks near θ=0.5)
  │        ╭╯     ╰╮
  │      ╭─╯       ╰─╮
  │   ╭──╯           ╰──╮
  │╭──╯                 ╰──╮
  └────────────────────────── θ
  -3  -2  -1   0   1   2   3
                  ↑
              b = 0.5

• Items too easy ($b_i \ll \hat{\theta}$): $P_i \to 1$, uncertainty term $\frac{1-P_i}{P_i} \to 0$, information drops
• Items too hard ($b_i \gg \hat{\theta}$): $P_i \to c_i$, guessing correction $(P_i-c_i)^2 \to 0$, information drops
• Items well-matched ($b_i \approx \hat{\theta}$): both terms are at reasonable values, information peaks

Concrete example at $\theta = 0$

Using $P$ values from the table above:

Item 5 (difficulty = 0.0, closest to $\theta = 0$) has the highest information at 0.338. This will be selected.

Task 6: Select Maximum Information Item

selected <- items[which.max(items$information), ]

which.max()

Returns the row index of the maximum value in a vector:

items$information <- c(0.013, 0.037, 0.080, 0.129, 0.338, 0.212, 0.082, 0.016, 0.002, 0.000)
which.max(items$information)   # → 5  (index of item 5)

This implements the Maximum Information selection criterion:

items[5, ]

Selects the entire row for item 5 — all columns (question, options, parameters):

selected <- items[5, ]
# A data frame with 1 row, all columns

The trailing comma is essential: items[5, ] selects row 5 (all columns). Without it, items[5] would select column 5 instead.

Task 7: Set Display Variables

question       <- as.character(selected$question)
option_a       <- as.character(selected$option_a)
option_b       <- as.character(selected$option_b)
option_c       <- as.character(selected$option_c)
option_d       <- as.character(selected$option_d)
correct_answer <- as.character(selected$correct_answer)
current_id     <- as.integer(selected$id)

Why as.character()?

When a column is extracted from a data frame row, R may return a factor level instead of a plain string, especially for character columns. Factor levels do not always behave like strings in template substitution:

# Without conversion:
selected$question   # → factor with level "What is 7x8?"
# Template: {{question}} might fail or show the factor level number

# With conversion:
as.character(selected$question)   # → "What is 7x8?"   plain string ✅

Why as.integer() for current_id?

current_id is used later in eval - score inside a SQL WHERE clause:

paste0("SELECT * FROM item_bank_3pl WHERE id = ", current_id)

If current_id is a string "5", this produces valid SQL: WHERE id = 5. But making it an integer explicitly ensures correct behavior and prevents potential quoting issues in edge cases:

as.integer("5")   # → 5L (integer)

These variables are passed to showPage - question via flow pointers

The {{variable}} placeholders in the HTML template are replaced by these variable values:

<div class="question">{{question}}</div>
<!-- becomes: -->
<div class="question">What is 7x8?</div>

Task 8: Track the Used Item

used_items <- c(used_items, current_id)

What it does

Appends current_id to the used_items vector, recording that this item has been administered.

Why this must happen in eval - select item not eval - score

The item ID must be tracked before the item is shown, not after it is scored. If the test-taker closes the browser mid-question (after seeing but before submitting), the eval - score node never runs. Tracking in eval - select item ensures the item is always recorded as used even in this edge case.

Progression across the test

# Before Q1: used_items = numeric(0)
# After Q1 selection (item 5):  used_items = c(5)
# After Q2 selection (item 6):  used_items = c(5, 6)
# After Q3 selection (item 3):  used_items = c(5, 6, 3)
# ...
# After Q10 selection (item 2): used_items = c(5, 6, 3, 7, 1, 9, 4, 8, 10, 2)

This vector is passed back via output flow variable pointer (↑) so the next iteration of eval - select item can filter it out.

How Item Selection Adapts Across the Test

The key insight is that theta changes after every item (updated by eval - score), and eval - select item always uses the latest theta. This creates the adaptive cycle:

Round 1: theta = 0.0  → select item with b closest to 0.0 → item 5 (b=0.0)
         Test-taker answers CORRECTLY
         EAP update: theta = 0.38

Round 2: theta = 0.38 → select item with max I(0.38) from remaining 9 items
         → item 6 (b=0.5) now optimal (closest to 0.38)
         Test-taker answers INCORRECTLY
         EAP update: theta = 0.15

Round 3: theta = 0.15 → select item with max I(0.15) from remaining 8 items
         → item 6 already used, next best is item 4 (b=-0.5) or item 5 (b=0.0)
         ...

The test converges on the true ability through this iterative approximation process.

Why Maximum Information is the Right Criterion

Fisher Information and theta estimation

The Fisher Information is the expected value of the squared score function:

For IRT, this simplifies to the Item Information Function $I_i(\theta)$. The Cramér-Rao lower bound states that no unbiased estimator can have variance smaller than:

Therefore, maximizing the total information $\sum_i I_i(\theta)$ minimizes the estimation variance. By selecting the item with the highest $I_i(\hat{\theta})$ at each step, CAT greedily minimizes the variance of the final theta estimate with each question.

Comparison with simpler alternatives

For our 10-item demonstration CAT, Maximum Information is appropriate. For operational CATs with security requirements, exposure control methods (Sympson-Hetter, randomesque) are preferred.

Node 3: showPage - question — Display Question

Concept

Presents the selected item to the test-taker. We use showPage (not form) because it reliably receives flow variables via Flow variable pointers.

Steps

1. Right-click canvas → showPage
2. Rename it to showPage - question
3. Click the node → find HTML field → click edit icon → paste:

<!DOCTYPE html>
<html>
<head>
  <style>
    * { box-sizing: border-box; margin: 0; padding: 0; }
    body {
      font-family: Arial, sans-serif;
      background: #f5f5f5;
      display: flex;
      justify-content: center;
      align-items: flex-start;
      min-height: 100vh;
      padding: 40px 20px;
    }
    .card {
      background: white;
      border-radius: 8px;
      box-shadow: 0 2px 8px rgba(0,0,0,0.1);
      padding: 32px;
      max-width: 640px;
      width: 100%;
    }
    .progress {
      font-size: 13px;
      color: #888;
      margin-bottom: 8px;
    }
    .progress-bar-bg {
      background: #e0e0e0;
      border-radius: 4px;
      height: 6px;
      margin-bottom: 24px;
    }
    .progress-bar-fill {
      background: #4a90e2;
      border-radius: 4px;
      height: 6px;
      width: calc({{answered}} / {{max_items}} * 100%);
    }
    .question {
      font-size: 18px;
      font-weight: bold;
      color: #333;
      margin-bottom: 24px;
      line-height: 1.5;
    }
    .options label {
      display: flex;
      align-items: center;
      padding: 12px 16px;
      border: 2px solid #e0e0e0;
      border-radius: 6px;
      margin-bottom: 10px;
      cursor: pointer;
      font-size: 15px;
      color: #444;
      transition: border-color 0.2s;
    }
    .options label:hover { border-color: #4a90e2; }
    .options input[type="radio"] { margin-right: 12px; }
    .btn-next {
      margin-top: 24px;
      width: 100%;
      padding: 14px;
      background: #4a90e2;
      color: white;
      font-size: 16px;
      border: none;
      border-radius: 6px;
      cursor: pointer;
    }
    .btn-next:hover { background: #357abd; }
  </style>
</head>
<body>
  <div class="card">
    <div class="progress">Question {{answered}} of {{max_items}}</div>
    <div class="progress-bar-bg">
      <div class="progress-bar-fill"></div>
    </div>
    <div class="question">{{question}}</div>
    <div class="options">
      <label><input type="radio" name="answer" value="A"> {{option_a}}</label>
      <label><input type="radio" name="answer" value="B"> {{option_b}}</label>
      <label><input type="radio" name="answer" value="C"> {{option_c}}</label>
      <label><input type="radio" name="answer" value="D"> {{option_d}}</label>
    </div>
    <button class="btn-next" type="submit">Next →</button>
  </div>
</body>
</html>

4. Click Save

Add input ports (↓)

Click blue + for each. Click each port → check Flow variable pointer → Pointed variable name → Save:

• question
• option_a
• option_b
• option_c
• option_d
• answered
• max_items

Add output port (↑)

Click red + → name it answer → click it → check Flow variable pointer → Pointed variable name: answer → Save

Connect

Drag from eval - select item out → showPage - question in

Node 4: eval - score — Score Response + Bayesian EAP Theta Estimation

Concept

This is the most important node. It performs three tasks after each response:

Task 1 — Score the response

Compare the submitted answer to correct_answer: $u_n = \begin{cases} 1 & \text{if answer is correct} \\ 0 & \text{if answer is incorrect} \end{cases}$

Task 2 — Update theta using Bayesian EAP

EAP treats theta estimation as a Bayesian inference problem [8]. After collecting $n$ responses $\mathbf{u} = (u_1, \ldots, u_n)$:

(a) Compute the likelihood of observing all responses at each grid point $\theta_k$:

For numerical stability, we compute the log-likelihood first then exponentiate:

(b) Apply the prior $\pi(\theta_k) = \phi(\theta_k)$ where $\phi$ is the standard normal PDF:

(c) Compute the posterior weights:

(d) Compute EAP estimate as the posterior mean:

(e) Compute the posterior standard error:

Task 3 — Check stopping rule

Why EAP is Better for CAT

• Works for all response patterns — including all-correct or all-incorrect (Newton-Raphson fails here) [8], [9]
• Provides SE — we know how precise our estimate is at each step [8]
• Stable — bounded by the prior; never diverges to ±∞ [10]
• Uses all response history — not just the last item [8]

Steps

1. Right-click canvas → eval
2. Rename it to eval - score
3. Click the node → edit Code field → paste:

# ── Task 1: Score the response ────────────────────────────────────────────────
is_correct    <- as.integer(answer == correct_answer)
total_correct <- total_correct + is_correct
answered      <- answered + 1

# ── Retrieve current item's IRT parameters from the database ──────────────────
current_item <- concerto.table.query(
  paste0("SELECT * FROM item_bank_3pl WHERE id = ", current_id)
)

a_new <- as.numeric(current_item$discrimination[1])
b_new <- as.numeric(current_item$difficulty[1])
c_new <- as.numeric(current_item$guessing[1])

# ── Append this item's parameters to history vectors ─────────────────────────
# These vectors grow by one entry per answered item
responses <- c(responses, is_correct)
items_a   <- c(items_a,   a_new)
items_b   <- c(items_b,   b_new)
items_c   <- c(items_c,   c_new)

# ── Task 2: Bayesian EAP Theta Estimation ─────────────────────────────────────
#
# We approximate the posterior integral using a discrete quadrature grid.
# The grid spans [-4, 4] with K=41 equally-spaced points.
# This range covers >99.99% of the standard normal prior.
#
# Formula:
#   θ̂_EAP = Σ(θ_k * w_k) / Σ(w_k)
#   SE_EAP = sqrt(Σ((θ_k - θ̂)² * w_k) / Σ(w_k))
#
# where w_k = L(θ_k | u) * π(θ_k)
#       L(θ_k | u) = likelihood of all responses at θ_k
#       π(θ_k)     = standard normal prior PDF

# Step 2a: Define the quadrature grid
K        <- 41                          # number of grid points
theta_grid <- seq(-4, 4, length.out = K)  # θ_k values

# Step 2b: Compute log-likelihood at each grid point
# For each θ_k, sum log-likelihoods across all answered items
log_lik <- numeric(K)

for (k in seq_len(K)) {
  th <- theta_grid[k]
  ll <- 0

  for (j in seq_len(answered)) {
    # 3PL probability for item j at grid point θ_k
    Pj <- items_c[j] + (1 - items_c[j]) / (1 + exp(-items_a[j] * (th - items_b[j])))

    # Safety clamp to avoid log(0)
    Pj <- max(min(Pj, 0.9999), 0.0001)

    # Add log-likelihood contribution of this item's response
    if (responses[j] == 1) {
      ll <- ll + log(Pj)
    } else {
      ll <- ll + log(1 - Pj)
    }
  }

  log_lik[k] <- ll
}

# Step 2c: Exponentiate log-likelihood (numerically stable via centering)
# Subtract max log-lik before exp to prevent overflow/underflow
log_lik_centered <- log_lik - max(log_lik)
likelihood       <- exp(log_lik_centered)

# Step 2d: Apply standard normal prior π(θ_k) = φ(θ_k)
prior   <- dnorm(theta_grid, mean = 0, sd = 1)

# Step 2e: Compute unnormalized posterior weights
weights <- likelihood * prior

# Step 2f: Normalize weights so they sum to 1
weights_sum      <- sum(weights)
weights_norm     <- weights / weights_sum

# Step 2g: EAP estimate — posterior mean
theta <- sum(theta_grid * weights_norm)

# Step 2h: Posterior standard error
se_theta <- sqrt(sum((theta_grid - theta)^2 * weights_norm))

# Round for display
theta    <- round(theta, 4)
se_theta <- round(se_theta, 4)

# ── Diagnostic log ────────────────────────────────────────────────────────────
cat("\n========================================\n")
cat(sprintf("EAP UPDATE after item %d\n", answered))
cat(sprintf("  Response:       %s (%s)\n",
    answer, ifelse(is_correct == 1, "CORRECT", "INCORRECT")))
cat(sprintf("  Item params:    a=%.3f, b=%.3f, c=%.3f\n", a_new, b_new, c_new))
cat(sprintf("  Response hist:  [%s]\n", paste(responses, collapse=",")))
cat(sprintf("  New theta (θ̂): %.4f\n", theta))
cat(sprintf("  SE(θ̂):         %.4f\n", se_theta))
cat(sprintf("  95%% CI:        [%.4f, %.4f]\n",
    theta - 1.96 * se_theta, theta + 1.96 * se_theta))
cat("========================================\n\n")

# ── Task 3: Fixed-length stopping rule ───────────────────────────────────────
# test_complete is a logical (TRUE/FALSE) passed to the if node
test_complete <- answered >= max_items