_            _    _        _         _
      /\ \         /\ \ /\ \     /\_\      / /\
      \_\ \       /  \ \\ \ \   / / /     / /  \
      /\__ \     / /\ \ \\ \ \_/ / /     / / /\ \__
     / /_ \ \   / / /\ \ \\ \___/ /     / / /\ \___\
    / / /\ \ \ / / /  \ \_\\ \ \_/      \ \ \ \/___/
   / / /  \/_// / /   / / / \ \ \        \ \ \
  / / /      / / /   / / /   \ \ \   _    \ \ \
 / / /      / / /___/ / /     \ \ \ /_/\__/ / /
/_/ /      / / /____\/ /       \ \_\\ \/___/ /
\_\/       \/_________/         \/_/ \_____\/
python-tokenizers 0.22.2
Dependencies: oniguruma@6.9.10
Channel: guix
Location: gnu/packages/machine-learning.scm (gnu packages machine-learning)
Home page: https://huggingface.co/docs/tokenizers
Licenses: ASL 2.0
Build system: pyproject
Synopsis: Implementation of various popular tokenizers
Description:

This package provides an implementation of today’s most used tokenizers, with a focus on performance and versatility.

Total packages: 1