It's not even about the cost of Prime or how long it takes to ship without it.
For me having Prime, I just bought way too much stuff on impulse. Like, paying for the service, $12 a month or whatever is. Is nothing. But then you feel like you have this service you need to be using it. If you arent using it, its throwing money away. So you end up buying way too much frivolous shit.
Now that I don't have it I only buy what I need. And in super rare cases I need something fast, just pay for faster shipping that one time and because im not paying $12 a month on prime, i'm still saving money if I pay for faster shipping those rare times it's needed.