English is not my first language. I suppose you would say "sweater" (but I said jacket because I prefer those garments with a zipper at the front: which I thought was always called "jacket" regardless of whether it's very light or thick).
You're not wrong. What people think will depend on context, and probably on region. In US English, jacket is a wide ranging term that could easily include a zip up sweater. If you wanted to be extra clear, you could saw "sweater jacket" or "fleece jacket" (if it's not woven).
