Skip to content

Inline plugin - iterating over items for album_fields significantly slows down operations #5773

Open
@BGarber42

Description

@BGarber42

Problem

Using even a simple album_field definition such as:

album_fields:
  album_bitrate: |
      total = 0
      for item in items:
          total += item.bitrate

significantly degrades performance.

Without:

root@6ea1b5cf64a4:/# time beet move -p 'title:Von dutch'
Moving 3 items.
[...]

real    0m2.702s
user    0m4.211s
sys     0m0.399s

With:

Moving 3 items.
[...]

real    0m15.705s
user    0m5.028s
sys     0m1.892s

With more items, it was significantly worse:

w/o:

root@6ea1b5cf64a4:/# time beet move -p 'artist:Charli XCX'
Moving 175 items.
[...]

real    0m5.264s
user    0m6.991s
sys     0m0.467s

with: (I actually gave up letting it run)

Moving 175 items.
[...]
^CTraceback (most recent call last):
  File "/lsiopy/lib/python3.12/site-packages/beets/ui/__init__.py", line 1870, in main
    _raw_main(args)
  File "/lsiopy/lib/python3.12/site-packages/beets/ui/__init__.py", line 1849, in _raw_main
    subcommand.func(lib, suboptions, subargs)
  File "/lsiopy/lib/python3.12/site-packages/beets/ui/commands.py", line 2216, in move_func
    move_items(
  File "/lsiopy/lib/python3.12/site-packages/beets/ui/commands.py", line 2146, in move_items
    objs = [o for o in objs if (isalbummoved if album else isitemmoved)(o)]
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lsiopy/lib/python3.12/site-packages/beets/ui/commands.py", line 2141, in isitemmoved
    return item.path != item.destination(basedir=dest)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lsiopy/lib/python3.12/site-packages/beets/library.py", line 1115, in destination
    subpath = self.evaluate_template(subpath_tmpl, True)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lsiopy/lib/python3.12/site-packages/beets/dbcore/db.py", line 701, in evaluate_template
    return t.substitute(
           ^^^^^^^^^^^^^
  File "/lsiopy/lib/python3.12/site-packages/beets/util/functemplate.py", line 559, in substitute
    res = self.interpret(values, functions)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lsiopy/lib/python3.12/site-packages/beets/util/functemplate.py", line 552, in interpret
    return self.expr.evaluate(Environment(values, functions))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lsiopy/lib/python3.12/site-packages/beets/util/functemplate.py", line 240, in evaluate
    out.append(part.evaluate(env))
               ^^^^^^^^^^^^^^^^^^
  File "/lsiopy/lib/python3.12/site-packages/beets/util/functemplate.py", line 177, in evaluate
    arg_vals = [expr.evaluate(env) for expr in self.args]
                ^^^^^^^^^^^^^^^^^^
  File "/lsiopy/lib/python3.12/site-packages/beets/util/functemplate.py", line 240, in evaluate
    out.append(part.evaluate(env))
               ^^^^^^^^^^^^^^^^^^
  File "/lsiopy/lib/python3.12/site-packages/beets/util/functemplate.py", line 145, in evaluate
    if self.ident in env.values:
       ^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen _collections_abc>", line 813, in __contains__
  File "/lsiopy/lib/python3.12/site-packages/beets/library.py", line 489, in __getitem__
    value = self._get(key)
            ^^^^^^^^^^^^^^
  File "/lsiopy/lib/python3.12/site-packages/beets/library.py", line 475, in _get
    return self._get_formatted(self.album, key)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lsiopy/lib/python3.12/site-packages/beets/dbcore/db.py", line 124, in _get_formatted
    value = model._type(key).format(model.get(key))
                                    ^^^^^^^^^^^^^^
  File "/lsiopy/lib/python3.12/site-packages/beets/dbcore/db.py", line 449, in _get
    return getters[key](self)
           ^^^^^^^^^^^^^^^^^^
  File "/lsiopy/lib/python3.12/site-packages/beetsplug/inline.py", line 120, in _func_func
    func.__globals__.update(_dict_for(obj))
                            ^^^^^^^^^^^^^^
  File "/lsiopy/lib/python3.12/site-packages/beetsplug/inline.py", line 102, in _dict_for
    out["items"] = list(obj.items())
                        ^^^^^^^^^^^
  File "/lsiopy/lib/python3.12/site-packages/beets/library.py", line 1295, in items
    return self._db.items(dbcore.MatchQuery("album_id", self.id))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lsiopy/lib/python3.12/site-packages/beets/library.py", line 1697, in items
    return self._fetch(Item, query, sort or self.get_default_item_sort())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lsiopy/lib/python3.12/site-packages/beets/library.py", line 1675, in _fetch
    return super()._fetch(model_cls, query, sort)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lsiopy/lib/python3.12/site-packages/beets/dbcore/db.py", line 1257, in _fetch
    rows = tx.query(sql, subvals)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/lsiopy/lib/python3.12/site-packages/beets/dbcore/db.py", line 956, in query
    cursor = self.db._connection().execute(statement, subvals)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt


real    5m15.560s
user    0m19.222s
sys     0m36.162s

Setup

  • OS: Unraid/Docker (linuxserver.io image)
  • Python version: Python 3.12.10
  • beets version: beets version 2.3.0
  • Turning off plugins made problem go away (yes/no): Yes

My configuration (output of beet config) is:

directory: /music/music
# --------------- Main ---------------

library: /config/library.db
pluginpath: /config/plugins

# --------------- Plugins ---------------

plugins:
- embedart
- fetchart
- info
- albumtypes
- lastgenre
- yearfixer
- fromfilename
- mbsync
- mbsubmit
- unimported
- zero
- missing
- inline
- badfiles
- substitute
- duplicates
- importreplace
- web
- originquery

# --------------- Performance ---------------

threaded: yes

# --------------- UI ---------------

verbose: no

# --------------- Tagging ---------------

per_disc_numbering: yes

import:
    log: /config/beet.log
    write: yes
    copy: no
    move: yes
    resume: yes
    incremental: yes
    incremental_skip_later: yes
    quiet: no
    quiet_fallback: skip
    from_scratch: no
    default_action: apply
    none_rec_action: ask
    duplicate_action: ask
    autotag: yes
art_filename: folder
mbcollection:
    auto: yes
    collection: XXXX
    remove: yes
musicbrainz:
    user: XXXX
    pass: XXXX
    extra_tags:
    - year
    - catalognum
    - country
    - media
    - label
zero:
    fields: comments
    auto: yes
    keep_fields: []
    update_database: no

# --------------- Import ---------------

clutter:
- Thumbs.DB
- Thumbs.db
- .DS_Store
- '**.zip'
- '**.torrent'
- '**.ini'
- '**.txt'
- '**.nfo'
- '**.m3u'
missing:
    format: $albumartist - $album - $title
    count: no
    total: no
    album: no

match:
    strong_rec_thresh: 0.08
    max_rec:
        missing_tracks: medium
        unmatched_tracks: medium
    preferred:
        countries:
        - US
        - XE
        - JP
        - GB|UK
        media: [Digital Media|File, CD, SACD]
        original_year: no
item_fields:
    is_mp3_aac: 1 if format == "MP3" or format == "AAC" else 0
    is_flac: 1 if format == "FLAC" else 0
    disc0: disc
    disctotal0: disctotal
album_fields:
    title0: title
albumtypes:
    types:
    -   ep: 'EP - '
    -   single: 'Single - '
    -   live: 'Live - '
    -   remix: 'Remix - '
    bracket: ''
    ignore_va: [compilation]
paths:
    default: '%substitute{$albumartist}/($original_year) $albumartist - %ifdef{altalbum,$altalbum,$album}%ifdef{transtitle, [$transtitle]} [%ifdef{disambig,$disambig - }%if{$atypes,$atypes}${format}%if{$is_mp3_aac, - $album_bitrate}%if{$is_flac, - $max_bitdepth - $max_samplerate}]/%if{$multidisc,Disc $disc0/}${track} - %ifdef{alttitle,$alttitle,$title}'
    collection::^.+: _Other/$collection/($original_year) $albumartist - %ifdef{altalbum,$altalbum,$album}%ifdef{transtitle, [$transtitle]} [%ifdef{disambig,$disambig - }${format}%if{$is_mp3_aac, - $album_bitrate}%if{$is_flac, - $max_bitdepth - $max_samplerate}]/%if{$multidisc,Disc $disc0/}${track} - %ifdef{alttitle,$alttitle,$title}
substitute:
    bladee &.*: Bladee
    blu &.*: Blu
    bob dylan &.*: Bob Dylan
    broadcast and.*: Broadcast
    casiopea.*: Casiopea
    czarface.*: Czarface
    "death\u2019s dynamic shroud and.*": "death\u2019s dynamic shroud"
    dizzy gillespie.*: Dizzy Gillespie
    drake &.*: Drake
    "el\u2010p feat.*": "El\u2010P"
    evaboy.*: miya lowe
    fred again...*: Fred again..
    .*gene clark with.*: Gene Clark
    gorillaz feat.*|spacemonkeyz &.*: Gorillaz
    "Hiromi\u2019s Sonicbloom|\u4E0A\u539F\u3072\u308D\u307F \u30B6\u30FB\u30C8\u30EA\u30AA\u30FB\u30D7\u30ED\u30B8\u30A7\u30AF\u30C8": Hiromi
    iglooghost.*: Iglooghost
    "jay\u2010z &.*": "JAY\u2010Z"
    john williams.*: John Williams
    jpegmafia.*: JPEGMAFIA
    "kanye west.*|kids see ghosts|\\\xA5\\$": Kanye West
    laufey with.*: Laufey
    macroblank &.*: Macroblank
    mf doom &.*|jj doom|dangerdoom|viktor vaughn|king geedorah|nehruviandoom|westsidedoom: MF DOOM
    neil young &.*: Neil Young
    quasimoto: Madlib
    rita lee &.*: Rita Lee
    round table.*: Round Table
    ryusenkei.*: Ryusenkei
    saint pepsi.*: SAINT PEPSI
    "Stan Getz & Jo\xE3o Gilberto.*": "Stan Getz & Jo\xE3o Gilberto"
    Sufjan Stevens &.*: Sufjan Stevens
    thaiboy digital &.*: Thaiboy Digital
    the moody blues.*: The Moody Blues
    the velvet underground.*: The Velvet Underground
    .*tommy heavenly.*: "Tommy february\u2076"
    trent reznor & atticus ross.*: Trent Reznor & Atticus Ross
    yung lean &.*: Yung Lean
    ^(.*?)( & Madlib| & The Alchemist).*: \1

importreplace:
    replacements: [{item_fields: artist artist_credit artists artists_credit composer, album_fields: artist artist_credit artists artists_credit composer, replace: {'Ye(?!\w)': Kanye West, 'Charli xcx(?!\w)': Charli XCX, 'RYUSENKEI(?!\w)': Ryusenkei}}, {item_fields: artist_sort artists_sort composer_sort, album_fields: artist_sort artists_sort composer_sort, replace: {'Ye(?!\w)': 'West, Kanye', 'Charli xcx(?!\w)': Charli XCX, 'RYUSENKEI(?!\w)': Ryusenkei}}]
fetchart:
    auto: no
    cover_names: folder cover front
    minwidth: 500
    maxwidth: 3000
    high_resolution: yes
    enforce_ratio: 0.5%
    sources:
    - filesystem
    - itunes
    - amazon
    - albumart
    -   coverart: releasegroup
    deinterlace: yes
    cover_format: JPEG
    quality: 98
    max_filesize: 0
    cautious: no
    store_source: no
    google_key: REDACTED
    google_engine: REDACTED
    fanarttv_key: REDACTED
    lastfm_key: REDACTED
embedart:
    auto: no
    maxwidth: 720
    quality: 95
    compare_threshold: 0
    ifempty: no
    remove_art_file: no
unimported:
    ignore_extensions: jpg png zip jepg log cue yaml db
    ignore_subdirectories: []
lastgenre:
    whitelist: /config/whitelist.txt
    auto: no
    canonical: yes
    count: 5
    fallback: ''
    force: yes
    min_weight: 10
    prefer_specific: no
    source: album
    title_case: yes
    separator: '; '
    keep_existing: no
    extended_debug: no
originquery:
    origin_file: origin.yaml
    tag_patterns:
        media: $.Media
        year: $."Edition year"
        label: $."Record label"
        catalognum: $."Catalog number"
        albumdisambig: $.Edition
web:
    host: 0.0.0.0
    port: 8337
    cors: ''
    cors_supports_credentials: no
    reverse_proxy: no
    include_paths: no
    readonly: yes
duplicates:
    album: no
    checksum: ''
    copy: ''
    count: no
    delete: no
    format: ''
    full: no
    keys: []
    merge: no
    move: ''
    path: no
    tiebreak: {}
    strict: no
    tag: ''
pathfields: {}
yearfixer:
    auto: no
    force: no
mbsubmit:
    format: $track. $title - $artist ($length)
    threshold: medium
    picard_path: picard

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions