You're missing the fact that "left" doesn't indicate "bigger", just semantics.
It's not as if values in memory appear left or right to each other.
Consider the following 32-bit integer 0x00112233:
- It's little-endian representation is "0x33 at the lowest address ... 0x00 at the highest address"
- It's big-endian representation is "0x00 at the lowest address ... 0x33 at the highest address"
In both cases, when you write it down in your code, you pad it with zeros on the left side.
Any compiler, whether it is designated for little-endian architecture or for big-endian architecture, will interpret the exact same value (0x112233).
But each type of compiler will generate a different memory layout for this value.
You could say that within that memory layout, the compiler will pad to the left in the case of little-endian and to the right in the case of big-endian.
However, this padding will have no impact on how you write it down in your code (it's not as if you'll need to change 0x00112233 to 0x33221100 when you switch from little-endian to big-endian).
And again - this is just semantics, because values in memory do not appear left or right to each other.
The only way for you to view the difference is by viewing the number from lowest byte-address to highest byte-address (or vice-versa).
For example, here is how you could do this in a native language like C:
int x = 0x12345678;
char* p = (char*)&x;
for (int i = 0; i < sizeof(x); i++)
As far as I know, you cannot do this type of "direct memory access" in Solidity.