Hi Nocash,
I'm not sure exactly how much detail of "How it works" you'd like, but I'll answer your questions and if I missed anything let me know:
Firstly, I think that having read the original documentation you now know what the AMX is, in short, it was a mouse which connected to the joystick port and moving the mouse sent the corresponding joystick up/down/left/right pulses to the CPC, while the three buttons sent the three possible fire signals (As you correctly stated, it more or less pulled the particular pin to 0V), but obviously in sequence with the polled common pin, otherwise any of the keyboard characters could appear instead of the joystick arrows, as the joystick common is being polled as part of the keyboard matrix, merely connecting the "up" pin to ground will give you the series of other keys which are in the same column of the matrix if it is grounded while the other column is being polled. The 74LS240 has an output enable pin, which is connected to the common, this means, the pulses are only passed on to the CPC at the correct moment.
A PS/2 mouse needs to be initialised and configured by a host before it will do anything useful and after this has been done, it sends back a 3 or 4 Byte serial signal on the data line (4 if in scroll-wheel mode). The first byte primarily contains 3 bits for the three button states and 2 bits for the sign (+/-) of the relative movements of the x and y axis, the second byte is a two's compliment value of the x movement and the third is the same for y movements. The fourth byte, which is only sent if the mouse has been switched to scroll-wheel mode contains the Z movement (ie: the scroll-wheel), the PIC in my circuit auto-detects the type of mouse and does all the required initialisation and then continuously polls the mouse and converts the two's compliment values into the associated up/down/left/right signals. It also has error handling, so that if the mouse is disconnected, it will wait for a new mouse and then re-initialise and re-start the polling.
Did you miss something important? No, probably not. The AMX software will work perfectly well with a 3 button joystick, but some people just like the idea of using the software with a mouse as it was originally intended. As well as that, modern CPC GUIs such as FutureOS and Symbos fully support AMX hardware and are much more comfortable to use with a mouse than a joystick.
The scroll wheel is a difficult one. Obviously it wasn't available on the original, so there is no standard to adhere to, but I thought, why waste a good scroll-wheel when it's there. As you also correctly stated (my confusing description of the scroll-wheel up/down signal) 6 is indeed up and 5 is down. The reason I swapped these, is because that's exactly what a PC scroll-wheel does, it swaps the concept of up and down, I could have just as easily used the signals the other way around, but I had to decide on one of them. To better explain the function: Rolling the scroll-wheel away from you (finger heading towards screen) will result in a 5, rolling your finger back towards yourself will result in a 6. If there's a compelling reason to reverse this, then let me know and I'd be happy to supply a new version of the firmware with the logic reversed. I will edit the project page and improve the description also.
The buttons are mapped the same as the AMX mouse (yes the original AMX documentation went out of its way to confuse the user): Assuming a right-handed mouse... The left button (index finger) will return an X, the right button will return a Z and the centre button is fire 3 (this one doesn't result in any character).
Does that explain it better? If I left anything out let me know...
Bryce.