You can't really do that easily without the ABI.
The first portion is the function selector, followed by the data to make the call.
The data which makes that call isn't so easily determined, as ABI encoding injects the dynamic parameters early in the calldata which specifies their offset and length. These easily can be confused for hardcoded parameters for things like address, or structs of deterministic types.
With the ABI, you can decode the data into they specified type.