💫 Enhancements and new features A new set of create-sibling-* commands reimplements the GitHub-platform support of create-sibling-github and adds support to interface three new platforms in a unified fashion: GIN (create-sibling-gin), GOGS (create-sibling-gogs), and Gitea (create-sibling-gitea). All commands rely on personal access tokens only for authentication, allow for specifying one of several stored credentials via a uniform --credential parameter, and support a uniform --dry-run mode for testing without network. #5949 (by @mih) create-sibling-github now has supports direct specification of organization repositories via a [<org>/]reposyntax #5949 (by @mih) create-sibling-gitlab gained a --dry-run parameter to match the corresponding parameters in create-sibling-{github,gin,gogs,gitea} #6013 (by @adswa) The --new-store-ok parameter of create-sibling-ria only creates new RIA stores when explicitly provided #6045 (by @adswa) The default performance of status() and diff() commands is improved by up to 700% removing file-type evaluation as a default operation, and simplifying the type reporting rule #6097 (by @mih) drop() and remove() were reimplemented in full, conceptualized as the antagonist commands to get() and clone(). A new, harmonized set of parameters (--what ['filecontent', 'allkeys', 'datasets', 'all'], --reckless ['modification', 'availability', 'undead', 'kill']) simplifies their API. Both commands include additional safeguards. uninstall is replaced with a thin shim command around drop() #6111 (by @mih) add_archive_content() was refactored into a dataset method and gained progress bars #6105 (by @adswa) The datalad and datalad-archives special remotes have been reimplemented based on AnnexRemote #6165 (by @mih) The result_renderer() semantics were decomplexified and harmonized. The previous default result renderer was renamed to generic. #6174 (by @mih) get_status_dict learned to include exit codes in the case of CommandErrors #5642 (by @yarikoptic) datalad clone can now pass options to git-clone, adding support for cloning specific tags or branches, naming siblings other names than origin, and exposing git clone's optimization arguments #6218 (by @kyleam and @mih) Inactive BatchedCommands are cleaned up #6206 (by @jwodder) export-archive-ora learned to filter files exported to 7z archives #6234 (by @mih and @bpinsard) datalad run learned to glob recursively #6262 (by @AKSoo) The ORA remote learned to recover from interrupted uploads #6267 (by @mih) A new threaded runner with support for timeouts and generator-based subprocess communication is introduced and used in BatchedCommand and AnnexRepo #6244 (by @christian-monch) A new switch allows to enable librarymode and queries for the effective API in use #6213 (by @mih) run and rerun now support parallel jobs via --jobs #6279 (by @AKSoo) A new foreach-dataset plumbing command allows to run commands on each (sub)dataset, similar to git submodule foreach #5517 (by @yariktoptic) The dataset parameter is not restricted to only locally resolvable file-URLs anymore #6276 (by @christian-monch) DataLad's credential system is now able to query git-credential by specifying credential type git in the respective provider configuration #5796 (by @bpoldrack) DataLad now comes with a git credential helper git-credential-datalad allowing Git to query DataLad's credential system #5796 (by @bpoldrack and @mih) The new runner now allows for multiple threads #6371 (by @christian-monch) A new configurationcommand provides an interface to manipulate and query the DataLad configuration. #6306 (by @mih) Unlike the global Python-only datalad.cfg or dataset-specific Dataset.config configuration managers, this command offers a uniform API across the Python and the command line interfaces. This command was previously available in the mihextras extension as x-configuration, and has been merged into the core package in an improved version. #5489 (by @mih) In its default dump mode, the command provides an annotated list of the effective configuration after considering all configuration sources, including hints on additional configuration settings and their supported values. The command line interface help-reporting has been sped up by ~20% #6370 #6378 (by @mih) ConfigManager now supports reading committed dataset configuration in bare repositories. Analog to reading .datalad/config from a worktree, blob:HEAD:.datalad/config is read (e.g., the config committed in the default branch). The support includes `reload() change detection using the gitsha of this file. The behavior for non-bare repositories is unchanged. #6332 (by @mih) The CLI help generation has been sped up, and now also supports the completion of parameter values for a fixed set of choices #6415 (by @mih) Individual command implementations can now declare a specific "on-failure" behavior by defining Interface.on_failure to be one of the supported modes (stop, continue, ignore). Previously, such a modification was only possible on a per-call basis. #6430 (by @mih) The run command changed its default "on-failure" behavior from continue to stop. This change prevents the execution of a command in case a declared input can not be obtained. Previously, only an error result was yielded (and run eventually yielded a non-zero exit code or an IncompleteResultsException), but the execution proceeded and potentially saved a dataset modification despite incomplete inputs, in case the command succeeded. This previous default behavior can still be achieved by calling run with the equivalent of --on-failure continue #6430 (by @mih) The `run command now provides readily executable, API-specific instructions how to save the results of a command execution that failed expectedly #6434 (by @mih) create-sibling --since=^ mode will now be as fast as push --since=^ to figure out for which subdatasets to create siblings #6436 (by @yarikoptic) When file names contain illegal characters or reserved file names that are incompatible with Windows systems a configurable check for save (datalad.save.windows-compat-warning) will either do nothing (none), emit an incompatibility warning (warning, default), or cause save to error (error) #6291 (by @adswa) Improve responsiveness of datalad drop in datasets with a large annex. #6580 (by @christian-monch) save code might operate faster on heavy file trees #6581 (by @yarikoptic) Removed a per-file overhead cost for ORA when downloading over HTTP #6609 (by @bpoldrack) A new module datalad.support.extensions offers the utility functions register_config() and has_config() that allow extension developers to announce additional configuration items to the central configuration management. #6601 (by @mih) When operating in a dirty dataset, export-to-figshare now yields and impossible result instead of raising a RunTimeError #6543 (by @adswa) Loading DataLad extension packages has been sped-up leading to between 2x and 4x faster run times for loading individual extensions and reporting help output across all installed extensions. #6591 (by @mih) Introduces the configuration key datalad.ssh.executable. This key allows specifying an ssh-client executable that should be used by datalad to establish ssh-connections. The default value is ssh unless on a Windows system where $WINDIR\System32\OpenSSH\ssh.exe exists. In this case, the value defaults to $WINDIR\System32\OpenSSH\ssh.exe. #6553 (by @christian-monch) create-sibling should perform much faster in case of --since specification since would consider only submodules related to the changes since that point. #6528 (by @yarikoptic) A new configuration setting datalad.ssh.try-use-annex-bundled-git=yes|no can be used to influence the default remote git-annex bundle sensing for SSH connections. This was previously done unconditionally for any call to datalad sshrun (which is also used for any SSH-related Git or git-annex functionality triggered by DataLad-internal processing) and could incur a substantial per-call runtime cost. The new default is to not perform this sensing, because for, e.g., use as GIT_SSH_COMMAND there is no expectation to have a remote git-annex installation, and even with an existing git-annex/Git bundle on the remote, it is not certain that the bundled Git version is to be preferred over any other Git installation in a user's PATH. #6533 (by @mih) run now yields a result record immediately after executing a command. This allows callers to use the standard --on-failure switch to control whether dataset modifications will be saved for a command that exited with an error. #6447 (by @mih) 🪓 Deprecations and removals The --pbs-runner commandline option (deprecated in 0.15.0) was removed #5981 (by @mih) The dependency to PyGithub was dropped #5949 (by @mih) create-sibling-github's credential handling was trimmed down to only allow personal access tokens, because GitHub discontinued user/password based authentication #5949 (by @mih) create-sibling-gitlab's --dryrun parameter is deprecated in favor or --dry-run #6013 (by @adswa) Internal obsolete Gitrepo.*_submodule methods were moved to datalad-deprecated #6010 (by @mih) datalad/support/versions.py is unused in DataLad core and removed #6115 (by @yarikoptic) Support for the undocumented datalad.api.result-renderer config setting has been dropped #6174 (by @mih) Undocumented use of result_renderer=None is replaced with result_renderer='disabled' #6174 (by @mih) remove's --recursive argument has been deprecated #6257 (by @mih) The use of the internal helper get_repo_instance() is discontinued and deprecated #6268 (by @mih) Support for Python 3.6 has been dropped (#6286 (by @christian-monch) and #6364 (by @yarikoptic)) All but one Singularity recipe flavor have been removed due to their limited value with the end of life of Singularity Hub #6303 (by @mih) All code in module datalad.cmdline was (re)moved, only datalad.cmdline.helpers.get_repo_instanceis kept for a deprecation period (by @mih) datalad.interface.common_opts.eval_default has been deprecated. All (command-specific) defaults for common interface parameters can be read from Interface class attributes (#6391 (by @mih) Remove unused and untested datalad.interface.utils helpers cls2cmdlinename and path_is_under #6392 (by @mih) An unused code path for result rendering was removed from the CLI main() #6394 (by @mih) create-sibling will require now "^" instead of an empty string for since option #6436 (by @yarikoptic) run no longer raises a CommandError exception for failed commands, but yields an error result that includes a superset of the information provided by the exception. This change impacts command line usage insofar as the exit code of the underlying command is no longer relayed as the exit code of the run command call -- although run continues to exit with a non-zero exit code in case of an error. For Python API users, the nature of the raised exception changes from CommandError to IncompleteResultsError, and the exception handling is now configurable using the standard on_failure command argument. The original CommandError exception remains available via the exception property of the newly introduced result record for the command execution, and this result record is available via IncompleteResultsError.failed, if such an exception is raised. #6447 (by @mih) Custom cast helpers were removed from datalad core and migrated to a standalone repository https://github.com/datalad/screencaster #6516 (by @adswa) The bundled parameter of get_connection_hash() is now ignored and will be removed with a future release. #6532 (by @mih) BaseDownloader.fetch() is logging download attempts on DEBUG (previously INFO) level to avoid polluting output of higher-level commands. #6564 (by @mih) 🐛 Bug Fixes create-sibling-gitlab erroneously overwrote existing sibling configurations. A safeguard will now prevent overwriting and exit with an error result #6015 (by @adswa) create-sibling-gogs now relays HTTP500 errors, such as "no space left on device" #6019 (by @mih) annotate_paths() is removed from the last parts of code base that still contained it #6128 (by @mih) add_archive_content() doesn't crash with --key and --use-current-dir anymore #6105 (by @adswa) run-procedure now returns an error result when a non-existent procedure name is specified #6143 (by @mslw) A fix for a silent failure of download-url --archive when extracting the archive #6172 (by @adswa) Uninitialized AnnexRepos can now be dropped #6183 (by @mih) Instead of raising an error, the formatters tests are skipped when the formatters module is not found #6212 (by @adswa) create-sibling-gin does not disable git-annex availability on Gin remotes anymore #6230 (by @mih) The ORA special remote messaging is fixed to not break the special remote protocol anymore and to better relay messages from exceptions to communicate underlying causes #6242 (by @mih) A keyring.delete() call was fixed to not call an uninitialized private attribute anymore #6253 (by @bpoldrack)