Solved Batch file character encoding problem

Jose Hidalgo · Apr 24, 2024

Hi everybody. I should mention that I'm based in France, so I have a French Win 11 (I guess, lol).

I have a .bat file that used to work perfectly under Win 7, and still does under Win 11, except when umlauts are involved.
For the record (but it's not relevant), the file uses ffmpeg's ffprobe binary to get the target video's width and height, and write them to a text file.

Here's a working example of the .bat file.

Code:

"C:\Program Files\FFmpeg\bin\ffprobe.exe" -v error -select_streams v:0 -show_entries stream=width,height -of csv=s=x:p=0 "D:\VIDEO\Clive Griffin & Celine Dion - When I Fall in Love.mkv" >"C:\Users\Jose\DOCUMENTS_D\Video\TAGGING\width & height\result.txt"

Here's a non-working example of the .bat file. Only one character differs.

Code:

"C:\Program Files\FFmpeg\bin\ffprobe.exe" -v error -select_streams v:0 -show_entries stream=width,height -of csv=s=x:p=0 "D:\VIDEO\Clive Griffin & Céline Dion - When I Fall in Love.mkv" >"C:\Users\Jose\DOCUMENTS_D\Video\TAGGING\width & height\result.txt"

CMD clearly points to an umlaut problem, due to the "é" in "Céline". You can try it yourself with any accented local file.

I've tried several things without success:

Several ffmpeg versions from 2023 and 2024
Several encodings of the .bat file (UTF-8, ANSI, and others), using Notepad and Notepad++.

How could I solve this annoying issue? Renaming file as "Celine" isn't an option since I have many other similar video files.

Thanks in advance!

Jose Hidalgo · Apr 24, 2024

Adding this at the beginning of the .bat file makes it work:

chcp 65001 (if the .bat file is in UTF-8 format)
chcp 1252 (if the .bat file is in ANSI format)

However, I'd like to avoid adding it every time, so it's not a real solution for the moment.

Typing chcp, I have found that my Win 11 CMD prompt is set to 850. I don't know if that's normal or not for a French Win 11.
See here: CHCP - Change Code Page - Windows CMD - SS64.com

However my Win 11 Notepad uses UTF-8 by default (which is weird BTW - my Win 7 Notepad used ANSI and I was happy with it).

So it would make sense to permanently set my CMD to 65001 via a Registry hack, right?...
... wrong, see here for the method and for details: Change permanently terminal encoding in windows 10

So, what should I do?...

iko22 · Apr 24, 2024

You can create another batch file that acts as a "wrapper script " for your batch file.

You don't say the name of your batch file, so I am going to call it mybatchfile.bat for the purpose of this example:

Create a wrapper batch file like this

Batch:

@ECHO OFF
REM Set the desired code page (e.g., UTF-8)
chcp 65001 > nul

REM Run your .bat file (replace "mybatchfile.bat" with your actual file name)
CALL mybatchfile.bat

(If your batch file has command line arguments, then you would need to add those to the wrapper script too and pass those arguments through to the CALL command.)

Then you would only write CHCP once, in one wrapper script batch file, which you can use many times.

Jose Hidalgo · Apr 24, 2024

Thanks! That sounds like a creative solution

It can work and at the same time avoid messing with the Registry. Its limitation is that the called batch file needs to have a fixed name. Right now my batch files have indexed names (e.g. myfile1.bat, myfile2.bat...), but I can adapt them and live with that. :thumbsup:

EDIT - another good solution would be a .bat file that automatically adds "chcp 65001" + CRLF at the beginning of any "myfile*.bat" file in the same directory.

I just don't understand why my scripts used to work flawlessly under Win 7, and now they don't with Win 11. Why Microsoft, why ???...

iko22 · Apr 25, 2024

Jose Hidalgo said:
...

EDIT - another good solution would be a .bat file that automatically adds "chcp 65001" + CRLF at the beginning of any "myfile*.bat" file in the same directory.

I just don't understand why my scripts used to work flawlessly under Win 7, and now they don't with Win 11. Why Microsoft, why ???...

The change to Notepad default encoding appears to have happened on Windows 10 version 1903. This happened in May 2019. Microsoft say the change was to "make it more compatible with other Microsoft applications." Although , I can see in your situation that is a disadvantage.

However, all is not lost, as quite possibility there is a registry change that can help which is different to the one you have already tried :

In Regedit go to Computer\HKEY_CURRENT_USER\Software\Microsoft\Notepad

- in the menu select edit/new/DWORD

- in the DWORD name enter iDefaultEncoding and enter value 1 as hexadecimal (it will automatically display as 0x0000001 (1) once you press OK)

[HKEY_CURRENT_USER\Software\Microsoft\Notepad]

"iDefaultEncoding"=dword:00000001

1 = ANSI ; 2 = UTF-16 LE ; 3 = UTF-16 BE ; 4 = UTF-8 BOM ; 5 = UTF-8

Source: Windows 10 1903) How to change Default Encoding UTF-8 to ANSI In Notepad?

See also How to Change the Default Character Encoding in Notepad

That would make Notepad behave more like your old Windows 7 Notepad.

Yes - you can also create a new batch file to automatically add CHCP 65001to the beginning of your bat files , if you wish. That would be another viable alternative.

Jose Hidalgo · Apr 25, 2024

Thanks again :) So Microsoft did change that on Win 10 v1903. I knew I wasn't crazy!

I'm happy to see that there's a registry hack to change Notepad's default encoding. But what would it solve? Like I said, I have already tried all Notepad encodings in the drop-down menu, and none of them solved my problem, UNLESS I changed CMD's encoding to either 65001 or 1252. It's like the default 850 encoding was incompatible with Notepad for all accented characters

That's what really puzzles me, and I'm happy to perform some testing if required.

I only see two solutions out of this:

Make Notepad behave like Win 7 Notepad AND also make CMD behave like Win 7 CMD
Don't touch the Notepad, but still make CMD behave in a way that's compatible with Win 11 Notepad

Maybe the suggested batch file is the easiest alternative after all :

A .bat file that automatically adds "chcp 65001" + CRLF at the beginning of any "myfile*.bat" file in the same directory

Any thoughts on how to achieve that? I have searched a bit online and I have found similar stuff, but nothing exactly like it.

Try3 · Apr 25, 2024

Jose Hidalgo said:
Make Notepad behave like Win 7 Notepad

Notepad is one of those Windows components that can be rescued from older Versions and re-used.
You do not have to try replacing the current one, you can just stick the rescued version anywhere you want and run it.
This post has a link to the proper procedure - my rescue components ditty - post #14 - ElevenForum

I've attached a zipped copy of the WinXP 32 bit version of Notepad.
It did not require the full procedure suggested by Ed.
You can copy it anywhere you want and just run it directly.
I do hope it does what you need.

Best of luck,
Denis

Jose Hidalgo · Apr 25, 2024

Hi Danis, thanks for chiming in.

Unfortunately, unless I'm wrong that would only solve a part of the problem:

Solution 1 : Make Notepad behave like Win 7 Notepad AND also make CMD behave like Win 7 CMD
or

Solution 2 : Don't touch the Notepad, but still make CMD behave in a way that's compatible with Win 11 Notepad

So if we can't do anything with the new CMD (for compatibility/stability reasons), then:

Either I keep adding "CHP 65001" manually to every new .bat file that I create, which is not very convenient,
or
I find a way to create a .bat file that automatically adds
Code:
```
"chcp 65001" + CRLF
```
at the beginning of any "myfile*.bat" file in the same directory.

I don't see another solution.

Try3 · Apr 26, 2024

Jose Hidalgo said:
that would only solve a part of the problem

Yes.
But, given that the problem did not exist with an earlier version, I thought that trying just the old Notepad might indeed solve the whole problem.

Denis

hdmi · Apr 26, 2024

Jose Hidalgo said:
A .bat file that automatically adds "chcp 65001" + CRLF at the beginning of any "myfile*.bat" file in the same directory

Put the batchfile below in the folder that contains the batchfiles all the names of which start with Mon fichier batch, and, in this same folder, create a new empty subfolder named Test. This batchfile reads all the batchfiles one by one, inserts CHCP 65001 on a separate new line at the beginning of each batchfile, and writes the modified batchfiles to the subfolder.

Batch:

@ECHO OFF
CHCP 65001>NUL
SET "A=Test"
PUSHD "%~dp0%A%" 2>NUL
IF ERRORLEVEL 1 (ECHO ERREUR - Le sous-répertoire ^"%A%^" n'existe pas. &GOTO End)
POPD
FOR %%I IN ("Mon fichier batch*.bat") DO (ECHO CHCP 65001 &TYPE "%%I")>"%~dp0%A%\%%I"
:End
ECHO Appuyez sur une touche pour quitter ^. ^. ^.
PAUSE >NUL

Berton · Apr 26, 2024

As for working with Win7, it may be there was more commonality with the older DOS commands, Win8 started more changes.

Jose Hidalgo · Monday at 8:45 AM

@hdmi, your test script works perfectly! Thank you

️

Now would it be possible to just have it replace the original .bat files in the same directory, without needing a test subfolder?
The cherry on top would be the script checking if the .bat files already start with CHCP 65001, and inserting it only for those who don't start with it.

Thanks again.

Tester · Monday at 11:55 AM

You could try to add chcp 65001 to the Autorun value of Command Processor.
Regkey: Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Command Processor
Make value REG_ZS name it as -> Autorun
Paste value: chcp 65001

Hope that works! Let me know

Jose Hidalgo · Monday at 12:48 PM

Tester said:
You could try to add chcp 65001 to the Autorun value of Command Processor.
Regkey: Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Command Processor
Make value REG_ZS name it as -> Autorun
Paste value: chcp 65001

Hope that works! Let me know

Thanks! Here's the thing. Like said earlier in this topic, apparently we should avoid permanent modifications to CMD, because there may be unexpected results. CMD should remain as it is, and CHCP 65001 should only be added when we need it.

I'll be perfectly happy with hdmi's script once it's refined, so we're almost there. Thanks again.

hdmi · Monday at 10:56 PM

I'm not 100% sure that it will always produce correct results, like, maybe if you try to use it to treat very long batchfiles it might fail. So, don't forget to create your backups.

Batch:

@ECHO OFF
CHCP 65001>NUL
SETLOCAL ENABLEDELAYEDEXPANSION
FOR %%I IN ("Mon fichier batch*.bat") DO (
  IF NOT "%%I"=="%~nx0" (
    CALL :Loop "%%I"
    IF /I "!C!"=="CHCP 65001" (
      REN "%%I" "%%I_"
      (ECHO !C!&TYPE "%%I_") >"%%I"
      DEL "%%I_"
    )
  )
)
GOTO End
:Loop
SET "C="
FOR /F "usebackq delims=" %%B IN ("%~1") DO (
  IF /I NOT "%%B"=="CHCP 65001" SET "C=CHCP 65001"
  GOTO :Eof
)
:End
ECHO Appuyez sur une touche pour quitter ^. ^. ^.
PAUSE >NUL

Jose Hidalgo · Wednesday at 5:33 PM

Works absolutely perfectly, even with reasonably long .bat files! Thank you so much @hdmi !

That's more than enough for me, so this topic is officially solved.
I hope this script may be of use to more people struggling with batch files and character codes.