# `token` --- Python 解析木で使われる定数

ソースコード: Lib/token.py

このモジュールは解析木の葉ノード(終端記号)の数値を表す定数を提供します。言語の文法のコンテキストにおける名前の定義については、 Python ディストリビューションのファイル file:Grammar/Tokens を参照してください。名前がマップする特定の数値は Python のバージョン間で変わります。

このモジュールは、数値コードから名前へのマッピングと、いくつかの関数も提供しています。関数は Python の C ヘッダファイルの定義を反映します。

Note that a token's value may depend on tokenizer options. For example, a `"+"` token may be reported as either `PLUS` or `OP`, or a `"match"` token may be either `NAME` or `SOFT_KEYWORD`.

token.tok_name

  ディクショナリはこのモジュールで定義されている定数の数値を名前の文字列へマップし、より人が読みやすいように解析木を表現します。

token.ISTERMINAL(x)

  終端トークンの値に対して `True` を返します。

token.ISNONTERMINAL(x)

  非終端トークンの値に対して `True` を返します。

token.ISEOF(x)

  x が入力の終わりを示すマーカーならば、 `True` を返します。

token の定数一覧:

token.NAME

  Token value that indicates an identifier or keyword.

token.NUMBER

  Token value that indicates a numeric literal

token.STRING

  Token value that indicates a string or byte literal, excluding formatted string literals. The token string is not interpreted: it includes the surrounding quotation marks and the prefix (if given); backslashes are included literally, without processing escape sequences.

token.OP

  A generic token value that indicates an operator or delimiter.

  CPython 実装の詳細: This value is only reported by the `tokenize` module. Internally, the tokenizer uses exact token types instead.

token.COMMENT

  Token value used to indicate a comment. The parser ignores `COMMENT` tokens.

token.NEWLINE

  Token value that indicates the end of a logical line.

token.NL

  Token value used to indicate a non-terminating newline. `NL` tokens are generated when a logical line of code is continued over multiple physical lines. The parser ignores `NL` tokens.

token.INDENT

  Token value used at the beginning of a logical line to indicate the start of an indented block.

token.DEDENT

  Token value used at the beginning of a logical line to indicate the end of an indented block.

token.FSTRING_START

  Token value used to indicate the beginning of an f-string literal.

  CPython 実装の詳細: The token string includes the prefix and the opening quote(s), but none of the contents of the literal.

token.FSTRING_MIDDLE

  Token value used for literal text inside an f-string literal, including format specifications.

  CPython 実装の詳細: Replacement fields (that is, the non-literal parts of f-strings) use the same tokens as other expressions, and are delimited by `LBRACE`, `RBRACE`, `EXCLAMATION` and `COLON` tokens.

token.FSTRING_END

  Token value used to indicate the end of a f-string.

  CPython 実装の詳細: The token string contains the closing quote(s).

token.TSTRING_START

  Token value used to indicate the beginning of a template string literal.

  CPython 実装の詳細: The token string includes the prefix and the opening quote(s), but none of the contents of the literal.

  Added in version 3.14.

token.TSTRING_MIDDLE

  Token value used for literal text inside a template string literal including format specifications.

  CPython 実装の詳細: Replacement fields (that is, the non-literal parts of t-strings) use the same tokens as other expressions, and are delimited by `LBRACE`, `RBRACE`, `EXCLAMATION` and `COLON` tokens.

  Added in version 3.14.

token.TSTRING_END

  Token value used to indicate the end of a template string literal.

  CPython 実装の詳細: The token string contains the closing quote(s).

  Added in version 3.14.

token.ENDMARKER

  Token value that indicates the end of input. Used in top-level grammar rules.

token.ENCODING

  ソースの bytes をテキストにデコードするために使われるエンコーディングを示すトークン値。`tokenize.tokenize()` は常に最初に ENCODING トークンを返します。

  CPython 実装の詳細: This token type isn't used by the C tokenizer but is needed for the `tokenize` module.

The following token types are not produced by the `tokenize` module, and are defined for special uses in the tokenizer or parser:

token.TYPE_IGNORE

  Token value indicating that a `type: ignore` comment was recognized. Such tokens are produced instead of regular `COMMENT` tokens only with the `PyCF_TYPE_COMMENTS` flag.

token.TYPE_COMMENT

  Token value indicating that a type comment was recognized. Such tokens are produced instead of regular `COMMENT` tokens only with the `PyCF_TYPE_COMMENTS` flag.

token.SOFT_KEYWORD

  Token value indicating a soft keyword.

  The tokenizer never produces this value. To check for a soft keyword, pass a `NAME` token's string to `keyword.issoftkeyword()`.

token.ERRORTOKEN

  Token value used to indicate wrong input.

  The `tokenize` module generally indicates errors by raising exceptions instead of emitting this token. It can also emit tokens such as `OP` or `NAME` with strings that are later rejected by the parser.

The remaining tokens represent specific operators and delimiters. (The `tokenize` module reports these as `OP`; see `exact_type` in the `tokenize` documentation for details.)

Token | 値
--- | ---

token.LPAR

`"("`

token.RPAR

`")"`

token.LSQB

`"["`

token.RSQB

`"]"`

token.COLON

`":"`

token.COMMA

`","`

token.SEMI

`";"`

token.PLUS

`"+"`

token.MINUS

`"-"`

token.STAR

`"*"`

token.SLASH

`"/"`

token.VBAR

`"|"`

token.AMPER

`"&"`

token.LESS

`"<"`

token.GREATER

`">"`

token.EQUAL

`"="`

token.DOT

`"."`

token.PERCENT

`"%"`

token.LBRACE

`"{"`

token.RBRACE

`"}"`

token.EQEQUAL

`"=="`

token.NOTEQUAL

`"!="`

token.LESSEQUAL

`"<="`

token.GREATEREQUAL

`">="`

token.TILDE

`"~"`

token.CIRCUMFLEX

`"^"`

token.LEFTSHIFT

`"<<"`

token.RIGHTSHIFT

`">>"`

token.DOUBLESTAR

`"**"`

token.PLUSEQUAL

`"+="`

token.MINEQUAL

`"-="`

token.STAREQUAL

`"*="`

token.SLASHEQUAL

`"/="`

token.PERCENTEQUAL

`"%="`

token.AMPEREQUAL

`"&="`

token.VBAREQUAL

`"|="`

token.CIRCUMFLEXEQUAL

`"^="`

token.LEFTSHIFTEQUAL

`"<<="`

token.RIGHTSHIFTEQUAL

`">>="`

token.DOUBLESTAREQUAL

`"**="`

token.DOUBLESLASH

`"//"`

token.DOUBLESLASHEQUAL

`"//="`

token.AT

`"@"`

token.ATEQUAL

`"@="`

token.RARROW

`"->"`

token.ELLIPSIS

`"..."`

token.COLONEQUAL

`":="`

token.EXCLAMATION

`"!"`

The following non-token constants are provided:

token.N_TOKENS

  The number of token types defined in this module.

token.EXACT_TOKEN_TYPES

  A dictionary mapping the string representation of a token to its numeric code.

  Added in version 3.8.

バージョン 3.5 で変更: Added `AWAIT` and `ASYNC` tokens.

バージョン 3.7 で変更: `COMMENT` 、`NL`、`ENCODING` トークンが追加されました。

バージョン 3.7 で変更: Removed `AWAIT` and `ASYNC` tokens. "async" and "await" are now tokenized as `NAME` tokens.

バージョン 3.8 で変更: Added `TYPE_COMMENT`, `TYPE_IGNORE`, `COLONEQUAL`. Added `AWAIT` and `ASYNC` tokens back (they're needed to support parsing older Python versions for `ast.parse()` with `feature_version` set to 6 or lower).

バージョン 3.12 で変更: Added `EXCLAMATION`.

バージョン 3.13 で変更: Removed `AWAIT` and `ASYNC` tokens again.