Do Serbian pupils know the score?

Q: Helen goes for a ride on her new bicycle. First, she rides four kilometers, which takes 10 minutes. Then, in five minutes she goes two kilometers. Did Helen go faster on the first trip, faster on the second trip, or equally fast? Or is it impossible to tell from the facts provided?

This is one of the problems more than 500,000 15-year-olds from around the world were asked to solve on the 2012 Program for International Student Assessment test, known as PISA. In the test’s six-point scale of difficulty, it was rated a two: answering correctly represented the minimum level to be considered “functionally numerate.” When Serbia’s PISA results were announced last month, questions like that of Helen’s cycling speed were the cause of much worry and puzzlement.

Historical building of the 3rd Gymnasium in Belgrade. Photo from Wikimedia Commons.

Forty percent of the 5,000 Serbian pupils who took the test were rated functionally innumerate. About one-third were rated functionally illiterate in reading. Of the 65 countries whose pupils were tested in three categories, Serbia ranked 43rd in math, 44th in reading, and 46th in science. While Serbian students scored two to seven points better on the triennial test than did their predecessors in 2009, the country’s overall ranking dropped one spot.

The poor showing received much media attention, with embarrassed educators and politicians promising better scores next time. But some education experts say the results themselves are only part of the problem. More worrying to them is what they say PISA reveals about how Serbia assesses its own students’ academic achievement.

By domestic standards, Serbian students perform exceedingly well. No less than one in seven is annually awarded the country’s highest honor in primary education: a “Vuk diploma,” named for the 19th-century linguist and folklorist Vuk Karadzic Stefanovic, the father of the modern Serbian alphabet. It is given to pupils who maintain straight A’s from fifth through eight grades and succeed in local and national academic competitions.

According to state figures, 35 percent of eighth-graders earned A’s in Serbian – reading, writing, and related subjects – in the 2012-2013 school year, and 27.4 percent got top marks in math.

Serbian schools, then, would seem to be producing excellent results. To those who view PISA as a truer indicator, the difference lies in the essentially opposite skills the international test and the classroom marks measure. It has led some to call for doing away with the prestigious Vuk diploma, which critics contend fosters an educational culture of cramming and grade inflation.

PISA is intended to show how well-equipped young people are for higher education and professional lives – or simply to live in modern society. Aleksandar Baucal, a psychology professor at the University of Belgrade and a member of the team that administers the test in Serbia, likens the bicycle question to situations people deal with daily, in shops, for example: if a liter of milk costs 90 euro cents and two liters is 1.80 euros, is one a better deal?

According to Baucal, 35 percent of Serbian pupils got the answer about Helen wrong – meaning they’d have trouble choosing the most economical milk purchase or solving similar problems in everyday life. Serbian schooling “hasn’t prepared them properly,” he said. “They are unable to recognize situations in life in which their knowledge can be useful, and to combine and creatively apply what they know.”

And the 77 percent of primary-school students who, based on their marks, are rated “excellent” or “very good” pupils, according to Serbia’s Institute for Education Quality and Evaluation? They are excellent or very good, the critics say – at doing what is asked of them. The problem is what the schools are asking.

In this view, studying, as practiced in Serbian schools, means cramming without understanding. Ivan Ivic, a retired psychology professor, spent his career in education and co-authored a 233-page strategy for schools through 2020 for the Education Ministry. The rote learning that predominates in primary and secondary education gives students “a short-lived, superficial knowledge that can seldom be used outside the specific school context,” he said.

Within that context, though, students earn high marks for providing exactly what is demanded, which “makes pupils feel it’s not important to develop their knowledge and skills,” Baucal said. “If that’s the approach to knowledge and learning nurtured in our schools, then the results of PISA shouldn’t be a surprise.”

Thus do many Serbian students end up “functionally illiterate” for today’s world – and the country itself illiterate for today’s economy, according to Ivic. That carries “huge long-term consequences for Serbia and its development, and for young people’s perspectives in a globalized world,” he said.

Analyzing university degrees in 2010, Ivic detected a dismal disconnect between what the education system provides and what the economy needs. Some 13 percent of graduates received various and often vaguely defined management degrees, far exceeding the output in “professional profiles that, because of the economic structure of Serbia, are more needed by the country,” he said.

As a particularly glaring example, Ivic cited food production, an area Serbian politicians have uniformly declared key to the country’s ability to compete in the European marketplace and boost employment. (“Our country is Europe’s green garden, and we must make use of that,” Prime Minister Ivica Dacic recently stated.) Just 1.6 percent of 2010 degrees were awarded by agricultural faculties.

A Closed Loop

Primary and secondary schools, however, are not judged on what students do after they leave. They gain status based on how many of their pupils are vukovci, recipients of Vuk diplomas, Ivic said – and they deliver grades accordingly. Teachers come under pressure to give high marks, from colleagues and administrators as well as from parents and pupils, who view all those A’s as tickets to future academic opportunities.

Within a school’s “closed system” of parents, students, and teachers, “everything is all right,” Ivic said. “The problem comes when leaving that system – when there’s a PISA or similar national test, and knowledge and abilities must be presented to the outside world.”

With the cramming model so entrenched, improvement requires change on a “systemic level,” said Dragica Pavlovic-Babic, the national PISA coordinator for Serbia. Policies and tests should be crafted to “demand abilities, not just reproduction of knowledge,” she said, and thus force “the entire teaching process to change focus.”

In Macedonia, where grades and standardized test scores also paint different pictures of student achievement, media reported late last month on steps aimed at fostering such a classroom shift. As of February, teachers who give grades that diverge widely from students’ test results will see their wages cut; instructors whose marks track with exam scores will get raises.

For some Serbian educators systemic change should start with abandoning the Vuk diploma, a notion floated recently by an association of Belgrade high schools. In a September letter to Education Minister Tomislav Jovanovic, the group said the award has “turned into the opposite of its intention, and as such harms the system of values in school.”

The letter followed a debate that simmered through the summer after the diploma gained outsized importance in determining sought-after secondary school placements – the result of the normal primary school final exam being scrapped because test questions were leaked.

Supporters of the idea say the award encourages rote learning and marked-up grades that do not reflect students’ true abilities. Pupils, vukovci included, would attain “a better quality of knowledge by learning through understanding, solving concrete problematic situations, applying knowledge,” Pavlovic-Babic said.

The proposal also generated much opposition. Belgrade daily Vecernje novosti quoted Desanka Radunovic, head of state-appointed policy body the National Education Council, as saying that abandoning the Vuk diploma “attacks the form and not the core” of the grading problem. Jovanovic told Serbian media he was “willing to discuss” the idea but cautioned against dropping the traditional honor too quickly.

“The Vuk diploma is a symbol representing more than its name implies. One shouldn’t make a decision on this overnight,” the minister said, adding that he would pursue efforts to “have children’s work properly evaluated, without the diploma being an issue.”

In Praise of Cramming

According to Pavlovic-Babic, students would see an immediate benefit from a shift in studying strategy. “Cramming bent over a book at home, memorizing [material] letter by letter, is hard and boring,” she said. Pupils could learn more in less time “by exchanging exhausting and ineffective ways of learning for effective ones.”

Cramming has its defenders, though. Aleksandar Lipkovski, president of the Mathematical Society of Serbia, dismissed as a “slogan” the notion that pupils can “study less and learn more.”

“A good education must include a certain amount of cramming,” he said. “To learn more, people must study more. The result is always proportional to the invested effort.”

Not only pedagogues share his view. Minja Jovanovic (no relation to the education minister) graduated from the University of Belgrade’s medical school last year and returned to her hometown of Smederevo, where she is interning at a local hospital. She has changed her formerly dim view of the learning-by-memorization that characterized her earlier schooling.

“I used to think that we were just cramming, but when I started university I realized that we do need that base of facts that we learn by heart,” she said. “At first it seems quite meaningless; only later, thanks to what you’ve crammed, are you able to connect things and start thinking for yourself.”

Jovanovic herself earned a Vuk diploma, which she recalled as “a great satisfaction. It was a reward and recognition for having spent all that energy through the years.” She believes the award motivates students to work harder. Parents of vukovci “are eager to attend the special parental meeting” where winners are honored, she added, “and to see their child’s photo at the school entrance,” where the recipients are often showcased as good examples for their fellow students.

The key problem in Serbian education, according to Lipkovski, isn’t the style of teaching or studying but the shift in ethics and values that have accompanied the transition from Yugoslavia’s one-party system to competing, and corrupting, political and social interests.

“A good education used to be available to all, and everyone knew that if he studied hard, success in life and in work would be guaranteed,” the mathematician said. Today, he maintained, young people believe a guaranteed job lies in “joining the party in power,” dampening their interest in a quality education.

Grade inflation and mass bestowing of Vuk diplomas are part of that moral devaluation, Lipkovski added, putting the blame largely on “parents who, at any price, want their children to have high marks, against all the criteria of real values.” But he also decries reliance on international mechanisms like PISA.

“The bad PISA results show that our school system hasn’t yet adapted to modern trends, with their pragmatic approach” of teaching to the test, he said. “Thank God our system hasn’t adapted!”

Some PISA advocates, like Baucal, also see a larger societal shadow over the pedagogical debate. He praised elements of the education strategy co-authored by Ivan Ivic, which cites as a major problem grading that is “indiscriminate and non-objective” and merely reflects students’ ability to regurgitate lessons. The document has been finished for nearly a year, but the Education Ministry has yet to act on its recommendations. Baucal suggested the delay might be due to Serbia lacking a consensus, “a common story,” about what it wants from education, which he terms the best tool Serbia has “to secure ourselves a better place in the world.”

“Our political elite,” Ivic said, “doesn’t realize that education is a basic development resource and not simply an expense.”

Tags: Research & quality, Students, Testing