I think if I was the driver I would just immediately note "customer declined package." If that somehow led to a job loss, I wouldn't be adverse to contacting a lawyer to sue any recipients that lied in feedback about their unreasonable request and the result, and/or Amazon.
I have seen the stories about Amazon being an unreasonable employer as far as employee requirements and/or feedback. But I've also seen just as many stories about them being perpetually short-handed on drivers and warehouse staff. It seems like there is already real natural pressure here for Amazon to not fire people for bad/wrong reasons, because making a bad habit of it makes one of their big challenges even worse.