The fact that silicon wafers are easy to break is a material property: the type of silicon used in semiconductor wafers is single-crystal silicon. That means all the atoms are carefully aligned into a single crystal form.
The wafers are sliced off of a single crystal of silicon pulled very slowly from a molten pool of ultra-pure silicon. One atom-sized impurity will cause a dislocation in the growth, throw off the atomic alignment of subsequent layers and cause the entire ingot to be rejected and scrapped. That is why silicon wafers are so expensive.
They are somewhat brittle and fairly easy to break but they won't fall apart in your hands. Normally, they are never touched with bare hands because the sodium in your perspiration will diffuse into the wafer and render them unusable.