Yes, both sound natural. Part of the reason for this is that one can read more than one newspaper or magazine, but one usually only reads one at a time. In addition, the plurals make sense if one is referring to what is read by all of the students collectively.

(1) Each student who is not reading a book is reading a newspaper or a magazine.
(2) Each student who is not reading books is reading newspapers or magazines.

(1) is more natural, although (2) is still correct. Here we are talking about what each student (singular) is reading, so the "what is being read by all of the students collectively" interpretation is ruled out. Also, the verb tense "is ... reading" focuses attention on what is happening right now, and right now it is unlikely that a single student is reading more than one newspaper or more than one magazine (although reading more than one book at a time is plausible when studying something). Thus "is reading a newspaper" is more likely than "is reading newspapers." However, "is ... reading" can also be used for what happens during an extended period of time including the present moment, so (2) is not impossible.