Dear Redwan
In my opinion I see that the Past Perfect is the optimum form for your second sentence
Because there are two events occurred in the past

It should be
In fact you didn't give me back the money I'd lent you

What you see ?