Q: How does the buy box algorithm know when is the limit I will not profit anymore- does it take it from the fees I have set in the default selling formula?

A: You create your minimum level of profitability by whatever formula the items are using (default or override). Yes, any values in your formula will be used to create your minimum.
For example, you want your minimum to be $3 profit + 15% amazon fees.  Then your formula would have
  • Fixed Margin $ = 3
  • Minimum Margin $ = 3
  • Amazon Fees % = 15

So if you're using that as your default formula in your amazon marketplace settings, and you have a buy box owner subscription and have it enabled for all items in your settings, and you do not have override formulas on the items, then buy box will allow your prices to stay in line with the competition unless their prices drop low to the point where they go below $3 take home profit + 15% for fees.  At that point, your item would stop there and not go lower. 

Note: You can use whatever values you want, this is just an example.