Some interesting quotes from the article:

“The current way we do AI puts a lot of burden on the designers to understand what the consequences of the incentives they give their systems are,” said Hadfield-Menell. “And one of the things we’re learning is that a lot of engineers have made mistakes.”

A major aspect of the problem is that humans often don’t know what goals to give our AI systems, because we don’t know what we really want.

Artificial Intelligence Will Do What We Ask. That’s a Problem. Quanta Magazine.