Reduce Remove() overhead because we don't have to walk the list to find an iterator by reference.
More robust and cancellable.