Fibres from the nasal half of each retina cross to the opposite side. Fibres from the temporal half stay where they are. Half the traffic changes lanes and half does not, which is the detail that makes the whole arrangement work.
The eye's optics invert the image, so the nasal retina views the temporal field and the temporal retina views the nasal field. Trace it through and the effect of a partial crossing is this: everything to the left of where you are looking, seen by either eye, ends up together in the right hemisphere, and everything to the right ends up in the left.
After the chiasm the brain is not organised by eye. It is organised by half of the world. That is what makes it possible to compare the two eyes' views of the same scene in one place, which is the raw material for stereoscopic depth.
It also means the two nerves have to meet, and where they meet matters. The chiasm sits directly above the pituitary gland. An anatomical coincidence, and the reason an endocrine tumour presents at an optician.