Computing systems are essentially made up of three types of devices: processors, memory, and interconnects. They have a somewhat fractal-like property, in that if you pick any piece, the piece tends to be composed of smaller, faster versions of these same devices.
The system block of a desktop computer can be assembled by hand with parts ordered over the internet or bought in a store.
Typical list of components:
The mainboard itself typically comes with many other components built-in: smaller processors for network, sound, peripherals, disks. These days CPUs also come with a small graphics processor built-in. But the primary purpose of the mainboard is as a place to house wires connecting all of these things.
When the power button is pushed, a small processor starts a simple operating system; after a chain of these, eventually the main operating system kernel is loaded and started on the main processor.
Devices are probed, device drivers (software for communicating with the device) are loaded, and at the end the kernel is able to read files and display graphics. Many devices require the kernel to upload software to them before they will work; this software is called firmware, and is typically sent to them by the kernel drivers. The rest is typically controlled by other programs which are loaded from the filesystem.