Invalidate cache on correct thread to avoid race