The code in these methods:

longlong Item::val_int_unsigned_typecast();

longlong Item::val_int_signed_typecast();

is not friendly to pluggable data types, as it tests for cmp_type().
Pluggable data types may want to implement a different way for signed/unsigned conversion.

Under terms of this task will add new methods in Type_handler:

virtual longlong Item_val_int_signed_typecast(Item *item) const;

virtual longlong Item_val_int_unsigned_typecast(Item *item) const;

and split the mentioned code into virtual implementations for relevant Type_handler_xxx classes.