Olive Oil with Lemon Cake Chunks

The first time I had olive oil ice cream someone had baked a lemon pound cake and just served it as an accompaniment, so the combination has always stuck with me. Why not just put the cake right into the ice cream?