You are incredibly uninformed. Middle/high school algebra is nowhere near as difficult as linear algebra. You need a calculus background to fully understand many aspects of it. The math progression is (generally) as follows:
middle school (or high school for some) algebra
high school geometry
Algebra 2 (which is more difficult than your algebra)
Precalculus (college algebra --much of what Algebra 2 is and Trigonometry)
Calculus I (differential calculus)
Calculus II (integral calculus)
Calculus III (vector calculus)
Discrete Math (introduction to proofs)
Linear Algebra I (using proofs)
Differential Equations
Linear Algebra II/other upper-division electives
Analysis/Abstract Algebra (this is where you learn why the assumptions you make in your high school - college algebra classes hold).
After this comes graduate-level courses. Here are some problems from a linear algebra I course for you to see: https://www.math.upenn.edu/ugrad/calc/m240/240la.pdf