Don’t throw the baby out with the bathwater is a popular phrase which means discarding something valuable and good when trying to get rid of something bad, or in other words rejecting something that is valuable to get rid of something that is worthless.
