From TeacherWiki

Professional Learning Communities enhanced by the ASSISTment online tutoring system

Cristina Heffernan, Professional Development coordinator for ASSISTment

Neil Heffernan , Creator and Principal Investigator of the ASSISTment system

Matt Millitillo, PIMSE Evaluator

Professional Learning Communities (PLC) have become commonplace in schools today as a remedy for underperformance. PLCs are intended to bring together teachers, principals, and other key school personnel to discuss and improve practice. The problem is that it is hard for a principal to get a PLC started. While the role of school principals allow them to reshape a school’s culture (Deal & Peterson, 1998) and to increase achievement (Leithwood et al., 2005; Marzano, et al., 2005), it is crucial that principals use the assistance of “math coaches” as an external change agent (West and Straub, 2003). A math coach (West & Fritz, 2003) is trained to mentor a PLC, establish the norms for teachers to reflect on their practices, get teachers to start visiting each other’s classrooms, and help them engage in all the positive aspects of a PLC. The principal needs to impart the vision of PLC to his staff, encourage their work, structure the school day to assist them, and hold the staff accountable for time spent in PLC activities. This is to say, PLCs cannot be content free. Effective PLCs focus on one problem of practice and then engage in a cycle of improvement (Militello, Rallis, & Goldring, 2009).

The following vignette attempts to explain the types of ways our teachers can use the ASSISTment system to help them change their teaching practices. While the vignette is fictional, we have based it upon some actual work we have seen with our own PLC in Worcester, Massachusetts. While reading it you can see our vision. Our goal, to raise student achievement by revolutionizing the way teachers work together in professional learning communities around math content, supported by a math data coach, and using the ASSISTment System as a tool. In our version of a PLC the coach is the external change agent, teaches routines – teachers determine common objective and assessment, then they all wind up giving the same lesson with one teacher is the lead. She presents to the PLC how they gave the lesson and show the data. Then as a group, they look at the data and talk about it. What would be an appropriate response to the data, stop and everyone looks and explore fully, then the lead tells them how she responded and possibly any new data collection. The process is both iterative and cyclical, involving all participants as both experts and learners.

Description of an ASSISTment and the ASSISTment system reports

Figure 1. A screen shot of an actual ASSISTment (#78) showing a student being tutored. From a student in Teacher A’s Class, see vignette.

Figure 1 shows a screenshot of a single ASSISTment question. This is a “main” problem combined with tutoring that a student receives when they get the initial problem wrong. A student who gets the main problem right will move on to another problem. For this ASSISTment question the student must know two skills: understanding Venn Diagrams and taking the Percent Of a number in order to get the problem right and move on. In this image we see that the student incorrectly typed “8,” to which the system responded, “Hmm, no. Let me break this down for you.” The tutor then asked the student a scaffolding question. In this case, the student must find the percent of students in Biology, Algebra and Band. We see that the student had to look at one hint to answer this question. This hint consists of an image to show that the student must add up the percents in gray. The next scaffolding question follows up on the first and this student got it correct. The last scaffolding question isolates the skill of Percent Of. This student typed in 23400 and was told that they did not check to see that her answer was reasonable (it must be less than 900), and was told that when multiplying by .27 they did not move the decimal over two points. Once the student actually types in the correct answer of 234 he or she proceeds to the next ASSISTment.

We call a problem on our system that has this type of scaffolding/tutoring an ASSISTment. We have just over 1000 such items in middle school math. Making good tutoring as shown in Figure 1 is hard and takes WPI on average 1.5 hours to create each ASSISTment with scaffolding questions and hints. Our authoring tool for this content is simple, and allows teachers the ability to add their own questions “naked” of any scaffolding and hint messages. Teachers have made thousands of these “naked” questions. Some teachers will write hints but not scaffolding questions of which we have about another thousand. Note how in this example, each child get individualized context-sensitive just-in-time-help, while the teacher does not have do to any grading. By breaking questions into steps, we are able to provide cognitively diagnostic information to teachers: which items are students getting right, which scaffold questions are they getting right, and what are common wrong answers. For example, the teacher of the student shown in Figure 1 knows the student does not understand the Venn diagram skill or the percent skill. This data is provided on the instant item report (see Figure 2).

Figure 2. This is an item report for a problem set in which item #78 (see Figure 1) was the last of four questions. This is data for Teacher A, see vignette.

Figure 2 shows a teacher an item report where his students have completed four questions in a row (items 4517, 8842, 4674 and 78). A teacher can see that over half of his students got the first question wrong but then seem to have learned from the tutoring and gotten the next two questions correct. The fourth item, ASSISTment #78, only 29% of student could get the item correct on their first attempt (if they have to ask for a hint they are marked wrong). The teacher sees that ASSISTment #78 also has the knowledge component “percent of” in addition to Venn Diagram. The teacher can see the common mistakes students made on the Original Question: 18% of his students answered 0, while 12% answered 657. If the teacher wants to find more detail on how students did on each scaffolding step, they just click on the link and are given a scaffolding report. For example, a teacher sees in Figure 3 where one column references the main question and then a column for each scaffolding question. With this report, a teacher can see which steps were the hardest, which students could handle the Venn Diagram steps, which could handle the percent, and which could complete both. The final column shows the results of a student’s open response that we will explain in the following vignette.

Figure 3. This is the report showing results from scaffolding questions and in the last column is the response to an open-ended question asked about this problem. This data is from Ms. D’s class – see vignette.

Vignette of Professional Learning Communities Focused on Formative Assessment

Ms. A is a member of a content-focused professional learning community (PLC) that is made up of four teachers including Ms. B, C, and D. They are all at different schools, but they all teach 7th grade math and have arranged to meet Tuesdays from 3-4 PM. On this particular Tuesday they are joined by Ms. Z, the math coach. Before they started their group they spent 3 days learning about the ASSISTment system, formative assessment and the expectations of working together as a group. Every other meeting they are joined by their coach Ms. Z who pushes them to reflect on the data they collect and helps them incorporate features of the ASSISTment system into their classroom practice. Ms. Z is also trying to get the teachers to do the math problems they are talking about instead of just skimming over them.

The most recent meeting ended with the teachers planning a lesson together that they would all teach during the week. In their group lesson plans, they always start with a learning objective, in this instance it was reading and solving problems with Venn Diagrams. Then they developed a lesson they will all teach and determine the assessment they will give. Before they met again they were expected to enact a complete formative assessment cycle. A formative assessment cycle begins with teaching the lesson, then giving the assessment and then responding to the collected data from the first assessment and finally assess again. They had learned from their coach that it is the second part of this cycle, responding to the data and re-assessing that makes the assessment a formative assessment. The assessment they picked this time was four ASSISTment questions that all had to do with Venn Diagrams. The last one on the assessment was ASSISTment #78 shown in Figure 1.

Now the teachers are meeting again. Their protocol is for one teacher, Ms. A this week, to be the “Lead teacher.” She recounted her lesson to the group. First she set up a large Venn diagram in the school parking lot. She had her students go out to the diagram and use their clothing to sort themselves into different circles. The first set of categories was: long sleeves, blue jeans and skirts. When they returned to class, she gave them the ASSISTment questions to see if they could solve problem using Venn Diagrams. They logged in and started working on the problems. Some students did not finish the assignment and Ms. A told them to finish it that night because she was going to look at the data first thing in the morning to help determine what they needed to work on the next day. The next day she took a look at the item report for the four items see Figure 2. Ms. A. has taught her students that most of the assessments she gives in class are intended to inform her instruction and their efforts in learning. Therefore they are not surprised when she posts the results of their work on the screen in their class first thing. They are also ready for her to respond to the data. She told her students how pleased she is at the over 85% success rate on the second and third problems especially after many of them struggled on the first question. The few students who got the second and third problems wrong, she assigned them Venn diagram mastery learning problem set for homework that will ensure they get practice until they reach mastery. There were two students who did not finish the problem set for homework and she informed their parents using the ASSISTment parent notification system.

Then she moved on to ASSISTment problem #78, (shown in Figure 1) where the results were not as good seen in the class represented in Figure 1, only 55% of the students got this right. As she brought the problem up on the screen she heard cries of “that one was hard.” That morning she had looked at the item report and noticed a common error. Looking at common wrong answers is always a bit of a puzzle. Why had students put “0” as an answer to this question? After some thought she realized that they thought all the percents in the three circles had to add up to 100%. Yikes! In this case there were people in the problem who did not fit into any of the circles. She decided to let the students who said “0” discuss what they learned about this Venn diagram by doing the scaffolding questions to these problems. She was curious to see if the tutoring they received online helped them clear up their misconception.

The other members of the PLC listened to Ms. A’s description of her class, her interpretation of her data and her response to the data. Now it was time for them to respond to her work. Ms. D was excited to hear why kids put “0” down as an answer as she had seen that common error in her student’s data but did not understand why kids said that. Ms. B and C both commented that their data was very similar to Ms. A’s but they responded to the data differently.

First Ms. B had asked the students who did not get the three first problems correct to come have lunch in her room so they could discuss Venn Diagrams again, they obviously needed some more instruction but with so many students showing proficiency she did not think she needed to address these concepts with the whole class again. For # 78 she dug a bit deeper into the data and looked at their responses from the scaffolding questions (see Figure 2 that shows students answers to the individual step)s. Once again students did well on the questions about the Venn Diagram. The place where they had trouble (only 47% correct) was the last step about taking a percent of 900 students. Ms. B was looking at the data in the morning, so she quickly wrote five questions on Percent Of like “What is 27% of 300?” in the ASSISTment system. She built these without tutoring but the students were told if they were right or wrong. The students who couldn’t go on because they couldn’t get the answer needed to get help and learn how to find the percent of a number. She was pleased to see them demonstrate leaning over the five questions and the percents correct kept going up.

Ms. C said that she did the lesson that day and has not responded to her data. She likes what B did so she asked her for the problem set number she made so she can use it the next day.

Ms. D reported that their coach Ms. Z had been in her class the day she did the lesson on Venn Diagrams. With both of them in the room, the students had finished the four question assessment earlier. They had been watching the data come in as the students worked and had also noticed how many of them had struggled with the percent problem (#78). They then quickly wrote an open response question. It asked, “Look at problem #78 again in another window using preview from your individual item report. Many people put 27 as the answer to the question. What mistake did they make?” Ms. D and the coach decided to only have half the students log into their accounts and have them compose the answer in pairs. They were able to clarify some misconceptions on Percent Of when they went over the questions. Ms. D was glad Ms. Z was there to help her orchestrate this questioning on the ASSISTment System and she liked getting all her students to write but she had never tried anything like this before (See Figure 3).

Speaking of getting help with ASSISTment features, Ms. C commented that at the next meeting when Ms. Z is there, she would like to talk to her more about Parent notification, she would like to start using that feature more, especially to praise the kids who are working so hard. Then they spent the last 15 minutes of their meeting designing the next lesson they would all do and determining the assessment to give on ASSISTments.

Summary of Vignette: Each of the teachers used the data provided by the ASSISTment System to customize the learning for their students. Teacher A used the cognitively diagnostic data provided by the ASSISTment System to inform her teaching. She assigned specialized homework, used mastery learning to get everyone up to speed and parental notification to help motivate students falling behind. Teacher B used the cognitively diagnostic data to decide to re-teach some material and used the Authoring tool to write her own questions. Teacher D used the advanced student response system to help her students understand Percent Of more clearly.