Can't mirror tensorflow properly for 3.11 when 3.12 is also configured

[alex-orange-UofU]

If I try to mirror tensorflow with python environments for both 3.11 and 3.12 then it won't grab tensorflow-io-gcs-filesystem which is required for python <3.12. From what I can gather the trouble is that there's no version of tensorflow-io-gcs-filesystem for 3.12 and so morgan gives up and decides not to get it. Not exactly sure why that's not fatal, but more to the point morgan should be handling the different environments somewhat more independently. Environment a needs package x, then get a version of package x that fulfills the requirement for environment a. Don't care if it doesn't work for environment b, you still need it for environment a. If you want to get fancy it ends up being a packing problem. I would suggest just doing the trivial "find a version that works for the environment I need it in" and don't worry about trying to reduce duplication.

[grische]

I think it might be interesting to have a flag to switch between "package needs to work in all environments" and "package needs to work in either environment":

https://github.com/ido50/morgan/blob/2c5ba7c45b80adb9ac57cdf93a0eb54469f242ad/morgan/__init__.py#L286-L290

[grische]

I think this means that #21 can be closed in favour of #46

EDIT: They are subtle different issues, see #21 (comment)

[alex-orange-UofU]

I'm not sure why. It's a duplicate of this one. I would just mark it as such and ask that this be fixed.

[alex-orange-UofU]

I think it might be interesting to have a flag to switch between "package needs to work in all environments" and "package needs to work in either environment":

morgan/morgan/init.py

Lines 286 to 290 in 2c5ba7c
for supported_python in self._supported_pyversions:
if not spec_set.contains(supported_python):
# file does not support the Python version of one of our
# environments, reject it
return False

I disagree. I see no value in trying to force package works in all environments. At best this maybe saves a tidbit of space, but it likely comes at the cost of having the most up to date package on newer versions of python. I would also say that if it fails to find one package that matches all versions, it then needs to fails (i.e. with an actual message). This is where the lack of value comes in for me, why have a system where failure is an option (as a consequence of choices made, not just lack of any version that could work) vs just getting separate versions for each version.

[grische]

I disagree. I see no value in trying to force package works in all environments.

I agree to you, and I am surprised this is no the default behaviour.

I still think it's best to have a flag to control the new behaviour as I would like to avoid breaking existing setups which assume exactly this "intersecting" behaviour.

Alternatively, one can of course also bump the major version to indicate a breaking change, but also this case should make a flag available to re-create the old behaviour.

[alex-orange-UofU]

Ok, I've read that one a bit more and the root issue is subtly different from mine. Mine is that morgan won't actually download resources sufficient to install on various different python versions. #46's issue is that it won't get a new version of the package that is available because it's already been covered, while for that case (Ansible) apparently it does still satisfy all the python versions requested.

[alex-orange-UofU]

I disagree. I see no value in trying to force package works in all environments.

I agree to you, and I am surprised this is no the default behaviour.

I still think it's best to have a flag to control the new behaviour as I would like to avoid breaking existing setups which assume exactly this "intersecting" behaviour.

I'm trying to understand the concern here. You don't want to give users a different version of the package in the morgan repo than they currently get? Presumably they're still usually going to get the same package because the intersection package is usually the newest version of package that works with the older python. Seems that if people depend on a very specific package existing they should be pinning the version of that package to make absolutely sure that they get it (given that of course the maintainers of the package can always put out a new version). What I'm saying is that I don't see the utility (besides storage space) of finding the intersection packages. If there is one I'd be interested to know what it is.

Alternatively, one can of course also bump the major version to indicate a breaking change, but also this case should make a flag available to re-create the old behaviour.

(edit, my comment showed up as part of the quote)

[grische]

It might be best to ask @ido50 what the original intention of the behaviour mentioned in #21 and #46 were.