You need to add a tag to the products you want to upsell. The tag is checkout-upsell.

If you have multiple products, the easiest way to add the tags is via the bulk edit → columns → Tags, add the tags to all the products you with easily.

We have a logic where it won’t offer a product as an upsell if that product is already in the cart.