fabro/checkpoint.json

{
  "timestamp": "2026-03-16T12:07:31.212736Z",
  "current_node": "extract_patch",
  "completed_nodes": [
    "start",
    "setup",
    "solve",
    "extract_patch"
  ],
  "node_retries": {
    "start": 1,
    "extract_patch": 1,
    "setup": 1,
    "solve": 4
  },
  "context_values": {
    "internal.fidelity": "compact",
    "internal.run_id": "01KKV8D40YW3216QWF0T0CGDD3",
    "internal.retry_count.setup": 1,
    "thread.solve.current_node": "extract_patch",
    "current.preamble": "Goal: In v5.3, NDDataRef mask propagation fails when one of the operand does not have a mask\n### Description\n\nThis applies to v5.3. \r\n\r\nIt looks like when one of the operand does not have a mask, the mask propagation when doing arithmetic, in particular with `handle_mask=np.bitwise_or` fails.  This is not a problem in v5.2.\r\n\r\nI don't know enough about how all that works, but it seems from the error that the operand without a mask is set as a mask of None's and then the bitwise_or tries to operate on an integer and a None and fails.\n\n### Expected behavior\n\nWhen one of the operand does not have mask, the mask that exists should just be copied over to the output.  Or whatever was done in that situation in v5.2 where there's no problem.\n\n### How to Reproduce\n\nThis is with v5.3.   With v5.2, there are no errors.\r\n\r\n```\r\n>>> import numpy as np\r\n>>> from astropy.nddata import NDDataRef\r\n\r\n>>> array = np.array([[0, 1, 0], [1, 0, 1], [0, 1, 0]])\r\n>>> mask = np.array([[0, 1, 64], [8, 0, 1], [2, 1, 0]])\r\n\r\n>>> nref_nomask = NDDataRef(array)\r\n>>> nref_mask = NDDataRef(array, mask=mask)\r\n\r\n# multiply no mask by constant (no mask * no mask)\r\n>>> nref_nomask.multiply(1., handle_mask=np.bitwise_or).mask   # returns nothing, no mask,  OK\r\n\r\n# multiply no mask by itself (no mask * no mask)\r\n>>> nref_nomask.multiply(nref_nomask, handle_mask=np.bitwise_or).mask # return nothing, no mask, OK\r\n\r\n# multiply mask by constant (mask * no mask)\r\n>>> nref_mask.multiply(1., handle_mask=np.bitwise_or).mask\r\n...\r\nTypeError: unsupported operand type(s) for |: 'int' and 'NoneType'\r\n\r\n# multiply mask by itself (mask * mask)\r\n>>> nref_mask.multiply(nref_mask, handle_mask=np.bitwise_or).mask\r\narray([[ 0,  1, 64],\r\n       [ 8,  0,  1],\r\n       [ 2,  1,  0]])\r\n\r\n# multiply mask by no mask (mask * no mask)\r\n>>> nref_mask.multiply(nref_nomask, handle_mask=np.bitwise_or).mask\r\n...\r\nTypeError: unsupported operand type(s) for |: 'int' and 'NoneType'\r\n```\r\n\n\n### Versions\n\n>>> import sys; print(\"Python\", sys.version)\r\nPython 3.10.11 | packaged by conda-forge | (main, May 10 2023, 19:07:22) [Clang 14.0.6 ]\r\n>>> import astropy; print(\"astropy\", astropy.__version__)\r\nastropy 5.3\r\n>>> import numpy; print(\"Numpy\", numpy.__version__)\r\nNumpy 1.24.3\r\n>>> import erfa; print(\"pyerfa\", erfa.__version__)\r\npyerfa 2.0.0.3\r\n>>> import scipy; print(\"Scipy\", scipy.__version__)\r\nScipy 1.10.1\r\n>>> import matplotlib; print(\"Matplotlib\", matplotlib.__version__)\r\nMatplotlib 3.7.1\r\n\n\n\n\n## Additional Context\n\nWelcome to Astropy 👋 and thank you for your first issue!\n\nA project member will respond to you as soon as possible; in the meantime, please double-check the [guidelines for submitting issues](https://github.com/astropy/astropy/blob/main/CONTRIBUTING.md#reporting-issues) and make sure you've provided the requested details.\n\nGitHub issues in the Astropy repository are used to track bug reports and feature requests; If your issue poses a question about how to use Astropy, please instead raise your question in the [Astropy Discourse user forum](https://community.openastronomy.org/c/astropy/8) and close this issue.\n\nIf you feel that this issue has not been responded to in a timely manner, please send a message directly to the [development mailing list](http://groups.google.com/group/astropy-dev).  If the issue is urgent or sensitive in nature (e.g., a security vulnerability) please send an e-mail directly to the private e-mail feedback@astropy.org.\n@bmorris3 , do you think this is related to that nddata feature you added in v5.3?\nHi @KathleenLabrie. I'm not sure this is a bug, because as far as I can tell the `mask` in NDData is assumed to be boolean: \r\n\r\nhttps://github.com/astropy/astropy/blob/83f6f002fb11853eacb689781d366be6aa170e0e/astropy/nddata/nddata.py#L51-L55\r\n\r\nThere are updates to the propagation logic in v5.3 that allow for more flexible and customizable mask propagation, see discussion in https://github.com/astropy/astropy/pull/14175.\r\n\r\nYou're using the `bitwise_or` operation, which is different from the default `logical_or` operation in important ways. I tested your example using `logical_or` and it worked as expected, with the caveat that your mask becomes booleans with `True` for non-zero initial mask values.\nWe are doing data reduction.  The nature of the \"badness\" of each pixel matters.  True or False does not cut it.  That why we need bits.  This is scientifically required.   A saturated pixel is different from a non-linear pixel, different from an unilliminated pixels, different .... etc. \r\n\r\nI don't see why a feature that had been there for a long time was removed without even a deprecation warning.\nBTW, I still think that something is broken, eg.\r\n```\r\n>>> bmask = np.array([[True, False, False], [False, True, False], [False, False, True]])\r\n>>> nref_bmask = NDDataRef(array, mask=bmask)\r\n>>> nref_bmask.multiply(1.).mask\r\narray([[True, None, None],\r\n       [None, True, None],\r\n       [None, None, True]], dtype=object)\r\n```\r\nThose `None`s should probably be `False`s not None's\nThere is *absolutely* a bug here. Here's a demonstration:\r\n\r\n```\r\n>>> data = np.arange(4).reshape(2,2)\r\n>>> mask = np.array([[1, 0], [0, 1]]))\r\n>>> nd1 = NDDataRef(data, mask=mask)\r\n>>> nd2 = NDDataRef(data, mask=None)\r\n>>> nd1.multiply(nd2, handle_mask=np.bitwise_or)\r\n...Exception...\r\n>>> nd2.multiply(nd1, handle_mask=np.bitwise_or)\r\nNDDataRef([[0, 1],\r\n           [4, 9]])\r\n```\r\n\r\nMultiplication is commutative and should still be here. In 5.2 the logic for arithmetic between two objects was that if one didn't have a `mask` or the `mask` was `None` then the output mask would be the `mask` of the other. That seems entirely sensible and I see no sensible argument for changing that. But in 5.3 the logic is that if the first operand has no mask then the output will be the mask of the second, but if the second operand has no mask then it sends both masks to the `handle_mask` function (instead of simply setting the output to the mask of the first as before).\r\n\r\nNote that this has an unwanted effect *even if the masks are boolean*:\r\n```\r\n>>> bool_mask = mask.astype(bool)\r\n>>> nd1 = NDDataRef(data, mask=bool_mask)\r\n>>> nd2.multiply(nd1).mask\r\narray([[False,  True],\r\n       [ True, False]])\r\n>>> nd1.multiply(nd2).mask\r\narray([[None, True],\r\n       [True, None]], dtype=object)\r\n```\r\nand, whoops, the `mask` isn't a nice happy numpy `bool` array anymore.\r\n\r\nSo it looks like somebody accidentally turned the lines\r\n\r\n```\r\nelif operand.mask is None:\r\n            return deepcopy(self.mask)\r\n```\r\n\r\ninto\r\n\r\n```\r\nelif operand is None:\r\n            return deepcopy(self.mask)\r\n```\r\n\n@chris-simpson I agree that line you suggested above is the culprit, which was [changed here](https://github.com/astropy/astropy/commit/feeb716b7412c477c694648ee1e93be2c4a73700#diff-5057de973eaa1e5036a0bef89e618b1b03fd45a9c2952655abb656822f4ddc2aL458-R498). I've reverted that specific line in a local astropy branch and verified that the existing tests still pass, and the bitmask example from @KathleenLabrie works after that line is swapped. I'll make a PR to fix this today, with a new test to make sure that we don't break this again going forward. \nMany thanks for working on this, @bmorris3.\r\n\r\nRegarding whether the `mask` is assumed to be Boolean, I had noticed in the past that some developers understood this to be the case, while others disagreed. When we discussed this back in 2016, however (as per the document you linked to in Slack), @eteq explained that the mask is just expected to be \"truthy\" in a NumPy sense of zero = False (unmasked) and non-zero = True (masked), which you'll see is consistent with the doc string you cited above, even if it's not entirely clear :slightly_frowning_face:.\nOf course I think that flexibility is great, but I think intentional ambiguity in docs is risky when only one of the two cases is tested. 😬 \nIndeed, I should probably have checked that there was a test for this upstream, since I was aware of some confusion; if only we could find more time to work on these important common bits that we depend on...\n\n## Completed stages\n- **setup**: fail\n  - Script: `git clone https://github.com/astropy/astropy.git . && git checkout b16c7d12ccbc7b2d20364b89fb44285bcbfede54 && sed -i 's/requires = \\[\"setuptools\",/requires = \\[\"setuptools==68.0.0\",/' pyproject.toml && python -m pip install -e .[test] --verbose`\n  - Stdout:\n    ```\n    fatal: destination path '.' already exists and is not an empty directory.\n    ```\n  - Stderr: (empty)\n- **solve**: fail\n\n## Context\n- failure_class: transient_infra\n- failure_signature: solve|transient_infra|api_transient|gemini|rate_limited\n",
    "outcome": "success",
    "thread.start.current_node": "setup",
    "internal.retry_count.start": 1,
    "internal.thread_id": "solve",
    "command.stderr": "",
    "internal.retry_count.extract_patch": 1,
    "internal.retry_count.solve": 4,
    "internal.node_visit_count": 1,
    "current_node": "extract_patch",
    "command.output": "",
    "graph.goal": "In v5.3, NDDataRef mask propagation fails when one of the operand does not have a mask\n### Description\n\nThis applies to v5.3. \r\n\r\nIt looks like when one of the operand does not have a mask, the mask propagation when doing arithmetic, in particular with `handle_mask=np.bitwise_or` fails.  This is not a problem in v5.2.\r\n\r\nI don't know enough about how all that works, but it seems from the error that the operand without a mask is set as a mask of None's and then the bitwise_or tries to operate on an integer and a None and fails.\n\n### Expected behavior\n\nWhen one of the operand does not have mask, the mask that exists should just be copied over to the output.  Or whatever was done in that situation in v5.2 where there's no problem.\n\n### How to Reproduce\n\nThis is with v5.3.   With v5.2, there are no errors.\r\n\r\n```\r\n>>> import numpy as np\r\n>>> from astropy.nddata import NDDataRef\r\n\r\n>>> array = np.array([[0, 1, 0], [1, 0, 1], [0, 1, 0]])\r\n>>> mask = np.array([[0, 1, 64], [8, 0, 1], [2, 1, 0]])\r\n\r\n>>> nref_nomask = NDDataRef(array)\r\n>>> nref_mask = NDDataRef(array, mask=mask)\r\n\r\n# multiply no mask by constant (no mask * no mask)\r\n>>> nref_nomask.multiply(1., handle_mask=np.bitwise_or).mask   # returns nothing, no mask,  OK\r\n\r\n# multiply no mask by itself (no mask * no mask)\r\n>>> nref_nomask.multiply(nref_nomask, handle_mask=np.bitwise_or).mask # return nothing, no mask, OK\r\n\r\n# multiply mask by constant (mask * no mask)\r\n>>> nref_mask.multiply(1., handle_mask=np.bitwise_or).mask\r\n...\r\nTypeError: unsupported operand type(s) for |: 'int' and 'NoneType'\r\n\r\n# multiply mask by itself (mask * mask)\r\n>>> nref_mask.multiply(nref_mask, handle_mask=np.bitwise_or).mask\r\narray([[ 0,  1, 64],\r\n       [ 8,  0,  1],\r\n       [ 2,  1,  0]])\r\n\r\n# multiply mask by no mask (mask * no mask)\r\n>>> nref_mask.multiply(nref_nomask, handle_mask=np.bitwise_or).mask\r\n...\r\nTypeError: unsupported operand type(s) for |: 'int' and 'NoneType'\r\n```\r\n\n\n### Versions\n\n>>> import sys; print(\"Python\", sys.version)\r\nPython 3.10.11 | packaged by conda-forge | (main, May 10 2023, 19:07:22) [Clang 14.0.6 ]\r\n>>> import astropy; print(\"astropy\", astropy.__version__)\r\nastropy 5.3\r\n>>> import numpy; print(\"Numpy\", numpy.__version__)\r\nNumpy 1.24.3\r\n>>> import erfa; print(\"pyerfa\", erfa.__version__)\r\npyerfa 2.0.0.3\r\n>>> import scipy; print(\"Scipy\", scipy.__version__)\r\nScipy 1.10.1\r\n>>> import matplotlib; print(\"Matplotlib\", matplotlib.__version__)\r\nMatplotlib 3.7.1\r\n\n\n\n\n## Additional Context\n\nWelcome to Astropy 👋 and thank you for your first issue!\n\nA project member will respond to you as soon as possible; in the meantime, please double-check the [guidelines for submitting issues](https://github.com/astropy/astropy/blob/main/CONTRIBUTING.md#reporting-issues) and make sure you've provided the requested details.\n\nGitHub issues in the Astropy repository are used to track bug reports and feature requests; If your issue poses a question about how to use Astropy, please instead raise your question in the [Astropy Discourse user forum](https://community.openastronomy.org/c/astropy/8) and close this issue.\n\nIf you feel that this issue has not been responded to in a timely manner, please send a message directly to the [development mailing list](http://groups.google.com/group/astropy-dev).  If the issue is urgent or sensitive in nature (e.g., a security vulnerability) please send an e-mail directly to the private e-mail feedback@astropy.org.\n@bmorris3 , do you think this is related to that nddata feature you added in v5.3?\nHi @KathleenLabrie. I'm not sure this is a bug, because as far as I can tell the `mask` in NDData is assumed to be boolean: \r\n\r\nhttps://github.com/astropy/astropy/blob/83f6f002fb11853eacb689781d366be6aa170e0e/astropy/nddata/nddata.py#L51-L55\r\n\r\nThere are updates to the propagation logic in v5.3 that allow for more flexible and customizable mask propagation, see discussion in https://github.com/astropy/astropy/pull/14175.\r\n\r\nYou're using the `bitwise_or` operation, which is different from the default `logical_or` operation in important ways. I tested your example using `logical_or` and it worked as expected, with the caveat that your mask becomes booleans with `True` for non-zero initial mask values.\nWe are doing data reduction.  The nature of the \"badness\" of each pixel matters.  True or False does not cut it.  That why we need bits.  This is scientifically required.   A saturated pixel is different from a non-linear pixel, different from an unilliminated pixels, different .... etc. \r\n\r\nI don't see why a feature that had been there for a long time was removed without even a deprecation warning.\nBTW, I still think that something is broken, eg.\r\n```\r\n>>> bmask = np.array([[True, False, False], [False, True, False], [False, False, True]])\r\n>>> nref_bmask = NDDataRef(array, mask=bmask)\r\n>>> nref_bmask.multiply(1.).mask\r\narray([[True, None, None],\r\n       [None, True, None],\r\n       [None, None, True]], dtype=object)\r\n```\r\nThose `None`s should probably be `False`s not None's\nThere is *absolutely* a bug here. Here's a demonstration:\r\n\r\n```\r\n>>> data = np.arange(4).reshape(2,2)\r\n>>> mask = np.array([[1, 0], [0, 1]]))\r\n>>> nd1 = NDDataRef(data, mask=mask)\r\n>>> nd2 = NDDataRef(data, mask=None)\r\n>>> nd1.multiply(nd2, handle_mask=np.bitwise_or)\r\n...Exception...\r\n>>> nd2.multiply(nd1, handle_mask=np.bitwise_or)\r\nNDDataRef([[0, 1],\r\n           [4, 9]])\r\n```\r\n\r\nMultiplication is commutative and should still be here. In 5.2 the logic for arithmetic between two objects was that if one didn't have a `mask` or the `mask` was `None` then the output mask would be the `mask` of the other. That seems entirely sensible and I see no sensible argument for changing that. But in 5.3 the logic is that if the first operand has no mask then the output will be the mask of the second, but if the second operand has no mask then it sends both masks to the `handle_mask` function (instead of simply setting the output to the mask of the first as before).\r\n\r\nNote that this has an unwanted effect *even if the masks are boolean*:\r\n```\r\n>>> bool_mask = mask.astype(bool)\r\n>>> nd1 = NDDataRef(data, mask=bool_mask)\r\n>>> nd2.multiply(nd1).mask\r\narray([[False,  True],\r\n       [ True, False]])\r\n>>> nd1.multiply(nd2).mask\r\narray([[None, True],\r\n       [True, None]], dtype=object)\r\n```\r\nand, whoops, the `mask` isn't a nice happy numpy `bool` array anymore.\r\n\r\nSo it looks like somebody accidentally turned the lines\r\n\r\n```\r\nelif operand.mask is None:\r\n            return deepcopy(self.mask)\r\n```\r\n\r\ninto\r\n\r\n```\r\nelif operand is None:\r\n            return deepcopy(self.mask)\r\n```\r\n\n@chris-simpson I agree that line you suggested above is the culprit, which was [changed here](https://github.com/astropy/astropy/commit/feeb716b7412c477c694648ee1e93be2c4a73700#diff-5057de973eaa1e5036a0bef89e618b1b03fd45a9c2952655abb656822f4ddc2aL458-R498). I've reverted that specific line in a local astropy branch and verified that the existing tests still pass, and the bitmask example from @KathleenLabrie works after that line is swapped. I'll make a PR to fix this today, with a new test to make sure that we don't break this again going forward. \nMany thanks for working on this, @bmorris3.\r\n\r\nRegarding whether the `mask` is assumed to be Boolean, I had noticed in the past that some developers understood this to be the case, while others disagreed. When we discussed this back in 2016, however (as per the document you linked to in Slack), @eteq explained that the mask is just expected to be \"truthy\" in a NumPy sense of zero = False (unmasked) and non-zero = True (masked), which you'll see is consistent with the doc string you cited above, even if it's not entirely clear :slightly_frowning_face:.\nOf course I think that flexibility is great, but I think intentional ambiguity in docs is risky when only one of the two cases is tested. 😬 \nIndeed, I should probably have checked that there was a test for this upstream, since I was aware of some confusion; if only we could find more time to work on these important common bits that we depend on...",
    "graph.rankdir": "LR",
    "thread.setup.current_node": "solve",
    "failure_class": "",
    "failure_signature": ""
  },
  "logs": [],
  "node_outcomes": {
    "start": {
      "status": "success",
      "duration_ms": 0
    },
    "solve": {
      "status": "fail",
      "failure": {
        "message": "LLM error: Rate limited by gemini: You exceeded your current quota, please check your plan and billing details. For more information on this error, head to: https://ai.google.dev/gemini-api/docs/rate-limits. To monitor your current usage, head to: https://ai.dev/rate-limit. \n* Quota exceeded for metric: generativelanguage.googleapis.com/generate_requests_per_model_per_day, limit: 250, model: gemini-3.1-pro\nPlease retry in 11h52m31.063433953s.",
        "failure_class": "transient_infra",
        "failure_signature": "api_transient|gemini|rate_limited"
      },
      "duration_ms": 300768
    },
    "setup": {
      "status": "fail",
      "context_updates": {
        "command.stderr": "",
        "command.output": "fatal: destination path '.' already exists and is not an empty directory.\n"
      },
      "failure": {
        "message": "Script failed with exit code: 128\n\n## stdout\nfatal: destination path '.' already exists and is not an empty directory.\n",
        "failure_class": "deterministic"
      },
      "duration_ms": 27
    },
    "extract_patch": {
      "status": "success",
      "context_updates": {
        "command.output": "",
        "command.stderr": ""
      },
      "notes": "Script completed: git diff",
      "duration_ms": 29
    }
  },
  "next_node_id": "exit",
  "loop_failure_signatures": {
    "setup|deterministic|script failed with exit code: <n> ## stdout fatal: destination path '.' already exists and is not an empty directory.": 1
  },
  "node_visits": {
    "solve": 1,
    "extract_patch": 1,
    "setup": 1,
    "start": 1
  }
}