Variable Name | [Short label] Full Wording of the Question and the Response Set |
|---|---|
q1_methodology | [Methodology] Throughout this course, how often have you done each of the following?... Describe the methods that you chose to solve a quantitative problem and why they were appropriate. |
q2_calculation | [Calculation]... Carried out calculations, either by hand or using software. |
q3_visual_representation | [Visual Representation]... Created a visual representation of numerical information, such as a graph, table, chart, or diagram. |
q4_interpretation | [Interpretation]... Described numerical information (numbers, graphs, statistics, etc.) by writing in words. |
q5_evaluation | [Evaluation]... Reached conclusions based on your own analysis of numerical information (numbers, graphs, statistics, etc.) |
q6_coherence_purpose | [Coherence/Purpose]... Constructed a single, coherent argument using several conclusions that were each based on quantitative evidence. |
q7_reasonableness | [Reasonableness]... Decided when a solution or inference was reasonable or not reasonable in the context of a problem. |
Quantitative Reasoning Indirect Assessment Report
1 Study
1.1 Background
A survey, Indirect Assessment in Quantitative Reasoning, was used to complement the Quantitative Reasoning Rubric for direct assessment.
1.2 Method and Sample
The survey was distributed through Google Forms to assess quantitative reasoning skills and engagement using the “Short Form.”
The data was collected during the last two weeks of the Spring 2024 semester, resulting in 469 complete student responses.
1.3 Variables
Variable names, full wording of the questions, short labels used in analyses, and the response sets are as follows.
2 Results
2.1 Quantitative Reasoning Skills: Summary
2.1.1 Table 1. Frequencies of Core Quantitative Reasoning Skills
Core QR Skills - Items | N = 4691 |
|---|---|
[Methodology] Described the methods used to solve a quantitative problem and why they were appropriate | |
Never | 19 (4.1%) |
Infrequently | 57 (12%) |
Sometimes | 202 (43%) |
Frequently | 188 (40%) |
[Calculation] Carried out calculations, either by hand or using software | |
Never | 9 (1.9%) |
Infrequently | 30 (6.4%) |
Sometimes | 125 (27%) |
Frequently | 304 (65%) |
[Visual Representation] Created a visual representation of numerical information (graph, table, chart, etc.) | |
Never | 14 (3.0%) |
Infrequently | 33 (7.0%) |
Sometimes | 157 (33%) |
Frequently | 265 (57%) |
[Interpretation] Described numerical information (numbers, graphs, statistics, etc.) by writing in words | |
Never | 12 (2.6%) |
Infrequently | 51 (11%) |
Sometimes | 151 (32%) |
Frequently | 252 (54%) |
[Evaluation] Reached conclusions based on your own analysis of numerical information (numbers, graphs, statistics, etc.) | |
Never | 10 (2.1%) |
Infrequently | 42 (9.0%) |
Sometimes | 175 (38%) |
Frequently | 239 (51%) |
[Coherence/Purpose] Constructed a single, coherent argument using several conclusions that were each based on quantitative evidence | |
Never | 27 (5.8%) |
Infrequently | 89 (19%) |
Sometimes | 190 (41%) |
Frequently | 159 (34%) |
[Reasonableness] Decided when a solution or inference was reasonable or not reasonable in the context of a problem | |
Never | 17 (3.6%) |
Infrequently | 53 (11%) |
Sometimes | 219 (47%) |
Frequently | 179 (38%) |
1n (%) | |
Table 1 presents the frequencies of student engagement in various quantitative activities within the class, measured through seven different questions.
“Carried out calculations, either by hand or using software,” shows that a significant majority of students, 65%, frequently engage in calculations. “Created a visual representation, like a graph, chart, or diagram,” indicates that 56.5% of students frequently create visual representations. “Interpreted numerical information by writing in words,” shows 54.1% of students frequently interpret numerical data in written form.
The last three items received the least “frequently” responses display more advanced quantitative reasoning and analysis.”Described the methods used to solve a quantitative problem and why they were appropriate” shows 40.3% of students frequently describe methodology. “Decided when a solution or inference was reasonable or not reasonable in the context of a problem” indicates that 38.2% of students frequently assess reasonableness. Finally, the item “Constructed a single, coherent argument using several conclusions that were each based on quantitative evidence” has the lowest “frequently” response at 34.2%, which is less common among students.
2.1.2 Figure 1. Frequencies of Core Quantitative Reasoning Skills
2.1.3 Table 2. Descriptive Statistics of Core Quantitative Reasoning Skills
Core QR Skills - Items | N = 4691 |
|---|---|
[Calculation] Carried out calculations, either by hand or using software | 3.55 (0.70) |
[Visual Representation] Created a visual representation of numerical information (graph, table, chart, etc.) | 3.43 (0.75) |
[Interpretation] Described numerical information (numbers, graphs, statistics, etc.) by writing in words | 3.38 (0.78) |
[Evaluation] Reached conclusions based on your own analysis of numerical information (numbers, graphs, statistics, etc.) | 3.38 (0.74) |
[Methodology] Described the methods used to solve a quantitative problem and why they were appropriate | 3.20 (0.81) |
[Reasonableness] Decided when a solution or inference was reasonable or not reasonable in the context of a problem | 3.20 (0.78) |
[Coherence/Purpose] Constructed a single, coherent argument using several conclusions that were each based on quantitative evidence | 3.03 (0.88) |
1Mean (SD) | |
Table 2 presents the descriptive statistics of core quantitative reasoning skills. The highest mean score is for “Calculation” at 3.55 out of four. “Visual representation” has a mean score of 3.43. “Interpretation” and “Evaluation” both have a mean score of 3.38. Students describe numerical information in written form and reach conclusions based on analysis at similar frequencies.
“Reasonableness” and “Methodology” both score 3.20. Students decide when solutions are reasonable in context and describe methods used to solve quantitative problems at moderate frequencies. “Coherence/Purpose,” which involves constructing coherent arguments using multiple quantitative conclusions, has a mean score of 3.03. Students construct coherent arguments using several conclusions based on quantitative evidence less frequently than other skills.
The standard deviations range from 0.70 to 0.88, with “Coherence/Purpose” showing the highest variability among students (SD = 0.88). In contrast, “Calculation” (SD = 0.70) and “Visual Representation” (SD = 0.75) show more consistent performance across students.
2.1.4 Figure 2. Descriptive Statistics of Quantitative Reasoning Skills
3 Correlation Analyses
3.1 Figure 3. Correlation Matrix of All Variables
Figure 9 presents the correlation matrix of all variables.
The strongest correlations within the core quantitative reasoning skills are observed between:
Coherence/Purpose and Reasonableness (r = 0.68, p<0.001)
Evaluation and Interpretation (r = 0.64, p<0.001)
Interpretation and Visual Representation (r = 0.61, p<0.001)
These strong correlations may suggest that students who build coherent arguments backed by quantitative evidence also tend to judge the reasonableness of solutions. Students with strong evaluation skills typically have strong interpretation abilities, and those who interpret numerical information also create visual representations of data.
3.2 Figure 4. Correlogram of All Variables
Note 2: Cells represent the correlation coefficient between two variables. Blue shows positive correlations, red shows negative correlations, and circle size reflects correlation magnitude. Correlation strength: weak (|r| < 0.3), moderate (0.3 < |r| < 0.5), strong (|r| > 0.5). Insignificant correlations are invisible (p > .05).
4 Regression Analyses
4.1 Table 3. Linear Regression Model Predicting Coherence/Purpose
Linear Regression Model Predicting Coherence/Purpose | ||||
|---|---|---|---|---|
Predictor | Estimate | SE | 95% CI | p-value |
Intercept | -0.01 | (0.03) | [-0.06, 0.05] | 0.837 |
Methodology | 0.21*** | (0.05) | [0.11, 0.30] | 0.000 |
Calculation | -0.11† | (0.06) | [-0.22, 0.00] | 0.053 |
Visual Representation | -0.05 | (0.05) | [-0.16, 0.05] | 0.310 |
Interpretation | 0.07 | (0.06) | [-0.04, 0.18] | 0.209 |
Evaluation | 0.24*** | (0.06) | [0.13, 0.35] | 0.000 |
Reasonableness | 0.53*** | (0.05) | [0.43, 0.63] | 0.000 |
SE = Standard Error, CI = Confidence Interval | ||||
A linear regression was conducted to identify the factors that influence students’ ability to construct a single, coherent argument using several conclusions based on quantitative evidence—referred to as “Coherence/Purpose.” This item received the lowest “frequently” responses at 34.2%, as noted in prior sections. Note that all variables were mean-centered prior to analyses to minimize multicollinearity.
The results reveal that four skills significantly influence the ability to construct coherent quantitative arguments. Specifically, the skill of describing the methods used to solve a quantitative problem and justifying their appropriateness increases the “Coherence/Purpose” score by 0.21 points on a 1-to-4 scale. Similarly, the ability to reach conclusions based on one’s own analysis of numerical information, such as numbers, graphs, or statistics, increases the score by 0.24 points. The most substantial impact arises from deciding when a solution or inference is reasonable within a problem’s context, which increases the Coherence/Purpose score by 0.53 points. Finally, “Carried out calculations, either by hand or using software” shows a negative effect, decreasing the Coherence/Purpose score by 0.11 points. This effect is significant at a marginal level (p < 0.10), a less reliable but noteworthy influence that warrants further exploration. The model explains approximately 51.1% of the variance in “Coherence/Purpose” (R² = 0.511).