Number of tokens limit in a FOR command in a Windows batch script

Question

Hello,

I was trying to process a text file in a Windows batch script and I ran into something that looks like a limitation to 31 tokens in a FOR loop. I isolated the issue in the code below:

@ECHO OFF
SET DATA=01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35

FOR /F "tokens=31* delims= " %%i IN ("%DATA%") DO (
    ECHO [%%i]
    ECHO [%%j]
)
ECHO.
FOR /F "tokens=32* delims= " %%i IN ("%DATA%") DO (
    ECHO [%%i]
    ECHO [%%j]
)

The output is:

[31]
[32 33 34 35]

[01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35]
[%j]

and I was expecting this:

[31]
[32 33 34 35]

[32]
[33 34 35]

Hoping that I haven't been doing something wrong, I couldn't find this limitation documented in the help for the FOR command. I'm using Windows XP. Do you know any workaround for this, aside from chopping off parts of the data?

Thank you.

Answer 1

From for /?:

%i is explicitly declared in the for statement and the %j and %k are implicitly declared via the tokens= option. You can specify up to 26 tokens via the tokens= line, provided it does not cause an attempt to declare a variable higher than the letter 'z' or 'Z'. Remember, FOR variables are single-letter, case sensitive, global, and you can't have more than 52 total active at any one time.

Answer 2

A token is the smallest unit of syntax that counts as one chunk. And a Batch command line in Windows 95/98/ME has a maximum limit of 64 tokens. Any more, and the command line will generate a Bad command or file name error.

which is why you're probably limited to 31 within DATA.

Answer 3

I came up with a solution. It's not elegant, but it solves my problem. When the commmand line interpreter cannot go further with the tokens, I pass the remaning of the data to a CALL :label command. Here is an example:

@ECHO OFF

SET DATA=01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100

FOR /F "tokens=1,31* delims= " %%i IN ("%DATA%") DO (
    ECHO  1st token: %%i
    ECHO 31th token: %%j
    CALL :processdatatokens32-62 %%k
)

:processdatatokens32-62
SET DATA=%*
FOR /F "tokens=1,31* delims= " %%i IN ("%DATA%") DO (
    ECHO 32nd token: %%i
    ECHO 62th token: %%j
    CALL :processdatatokens63-83 %%k
)
GOTO :EOF

:processdatatokens63-83
SET DATA=%*
FOR /F "tokens=1,31* delims= " %%i IN ("%DATA%") DO (
    ECHO 63th token: %%i
    ECHO 93th token: %%j
)
GOTO :EOF

The output is:

 1st token: 01
31th token: 31
32nd token: 32
62th token: 62
63th token: 63
93th token: 93

Answer 4

There is more than one way to loop through the DATA array:

@ECHO OFF
setlocal ENABLEDELAYEDEXPANSION
SET DATA=01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101

for /l %%a in (0,3,150) do @echo %%a data: !data:~%%a,2!

This is possible only because of the two character variables in the %DATA% array. As you will see, the limit is now 99 characters, rather than 31. When you get to the 100th and beyond, the number is truncated.

Rob