I have to agree with everyone who has mentioned external hard drives or flash drives. I HIGHLY recommend both. If you get a TA position, and if your department is like mine, you will be required to keep copies of all of your students' papers for two years (in case someone decides to challenge a grade, or if there are plagiarism accusations). To put this into perspective, look at my first semester as a TA: I taught comp 1, which requires five essays, ranging in length from 2-3 to 5-6 pages. Each student should therefore have written a minimum of 15 pages worth of essays. Multiply that by about 40-45 students. Now multiply that by four (semesters). Total: 2400 pages that I'm required to keep track of at any given time. Digital copies are a life (and space) saver. Flash drives are also good for running portable apps (portableapps.com), like office software, .pdf readers, web browsers, etc.
I also agree with the scanner idea, but, as others have already pointed out, you should wait to see if your department has one available for TA use. The English TAs at my school have their own computer/printer lab, with access to scanners, including a fancy one that automatically converts the scanned pages to a .pdf file, which is great for distributing digital handouts to students by email or through an online course manager.
And you should definitely get chalk and dry erase markers if you're going to be a TA. I learned very quickly that you can't count on the classroom to have a supply of them; people either take them or other instructors "borrow" them because there weren't any in their rooms either. For the chalk, you should also consider getting a chalk holder...it will save your fingers (and your clothes) from getting dusty, and it also cuts down on broken chalk.
Regardless, you should wait on the more "specialized" office supplies until you get to school. Your department may provide a lot of things for you. Incoming TAs in my dept. receive goodie bags filled with legal pads, pens, pencils, markers, chalk, staplers...all the basics that you could possibly need as a TA or student. Sometimes there are even flash drives or other more "techy" items in the bag.